@aiassesstech/mighty-mark 0.5.3 → 0.5.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +71 -0
- package/agent/AGENTS.md +2 -1
- package/agent/IDENTITY.md +1 -1
- package/agent/SOUL.md +2 -2
- package/dist/index.d.ts +1 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -1
- package/dist/index.js.map +1 -1
- package/package.json +1 -1
- package/src/scripts/fleet-bus-diagnostics.sh +433 -0
- package/src/scripts/seed-agent-memory.sh +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,77 @@ All notable changes to `@aiassesstech/mighty-mark` will be documented in this fi
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [0.5.5] — 2026-03-08
|
|
9
|
+
|
|
10
|
+
**Fleet-bus diagnostics script + updated VPS smoke test.**
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- `fleet-bus-diagnostics.sh` — command-line fleet-bus health checker (cards, audit chain, message flow, gateway init, live ping test).
|
|
14
|
+
- Supports `--live` (gateway ping test), `--verbose` (detailed event listing).
|
|
15
|
+
|
|
16
|
+
### Fixed
|
|
17
|
+
- Updated `vps-smoke-test.mjs` to use current `enforceRouting(FleetMessage)` API (was using outdated 3-arg signature).
|
|
18
|
+
- Added all 6 agent cards to smoke test (was only checking 4).
|
|
19
|
+
- Fixed `AuditTrailWriter` usage to use `.record()` instead of `.write()`.
|
|
20
|
+
|
|
21
|
+
## [0.5.4] — 2026-03-07
|
|
22
|
+
|
|
23
|
+
**Documentation sync — updated check counts, rules, guides, and deployment playbook.**
|
|
24
|
+
|
|
25
|
+
### Changed
|
|
26
|
+
- Updated Mark's agent files (SOUL.md, AGENTS.md, IDENTITY.md) to reflect 72 checks across 9 categories (was "26 across 5").
|
|
27
|
+
- Fixed seed-agent-memory.sh category count (9, not 8).
|
|
28
|
+
- Updated SPEC Section 6 (Mighty Mark Integration Mapping) — marked 4 checks as Built, added `checkTelegramAllowlist`.
|
|
29
|
+
- Updated Rule 806 — deployment scripts now reference npm package paths, added hardening scripts.
|
|
30
|
+
- Updated Deployment Playbook — added "Post-Install Hardening (New VPS)" onboarding flow.
|
|
31
|
+
- Updated CHANGELOG with entries for v0.5.0–v0.5.3.
|
|
32
|
+
- Updated trust-alliance and yellow-brick-road agent docs content.
|
|
33
|
+
|
|
34
|
+
## [0.5.3] — 2026-03-07
|
|
35
|
+
|
|
36
|
+
**Optimized agent memory seed script.**
|
|
37
|
+
|
|
38
|
+
### Added
|
|
39
|
+
- **`seed-agent-memory.sh`** — deploys role-specific MEMORY.md to all 6 fleet agents. Each agent gets operational knowledge tailored to their function (fleet structure, communication patterns, key decisions, lessons learned). Supports `--dry-run` and `--force` (backs up before overwriting).
|
|
40
|
+
- Also creates Jessie's HEARTBEAT.md (with cron + memory maintenance) if missing.
|
|
41
|
+
|
|
42
|
+
## [0.5.2] — 2026-03-07
|
|
43
|
+
|
|
44
|
+
**Deploy-fleet script bundled into npm package.**
|
|
45
|
+
|
|
46
|
+
### Added
|
|
47
|
+
- **`deploy-fleet.sh`** moved into `src/scripts/` — ships with the npm package for VPS deployment.
|
|
48
|
+
|
|
49
|
+
## [0.5.1] — 2026-03-07
|
|
50
|
+
|
|
51
|
+
**Telegram config structure fix.**
|
|
52
|
+
|
|
53
|
+
### Fixed
|
|
54
|
+
- **`checkTelegramAllowlist`** now correctly handles `channels.telegram` with per-account policies (not just top-level `config.telegram`). Supports `allowlist` and `pairing` DM policies.
|
|
55
|
+
- Extended `OpenClawConfig` interface to include `channels.telegram.accounts` structure.
|
|
56
|
+
- Updated unit tests and scenario tests for the new config path.
|
|
57
|
+
|
|
58
|
+
## [0.5.0] — 2026-03-07
|
|
59
|
+
|
|
60
|
+
**Post-install hardening checks + VPS deployment scripts.**
|
|
61
|
+
|
|
62
|
+
### Added
|
|
63
|
+
- **4 new security checks** in `security-hardening.ts` (total security checks: 8):
|
|
64
|
+
- `checkModelDefaults` — verifies primary model + at least one fallback
|
|
65
|
+
- `checkSecretPermissions` — verifies `~/.openclaw/secrets/` dir=700, file=600
|
|
66
|
+
- `checkTelegramAllowlist` — verifies DM policy is `allowlist` or `pairing`
|
|
67
|
+
- `checkWorkspacePersonalization` — verifies SOUL.md + IDENTITY.md exist and are customized (>100 bytes) for all 6 agents
|
|
68
|
+
- **`verify-post-install-hardening.sh`** — read-only VPS verification script (human/JSON/quiet output modes)
|
|
69
|
+
- **`apply-post-install-hardening.sh`** — applies hardening config (secrets dir, MEMORY.md, HEARTBEAT.md, openclaw.json patches). Supports `--dry-run` and `--skip-restart`.
|
|
70
|
+
- 56 unit tests for the 4 new security checks.
|
|
71
|
+
- Updated scenario tests with model defaults and channels.telegram config.
|
|
72
|
+
- Total morning check: 72 checks across 9 categories (gateway, agents, system, api, data, memory, fleet, alerting, security).
|
|
73
|
+
|
|
74
|
+
### Deployment Notes
|
|
75
|
+
- Copy new scripts from `src/scripts/` to `/opt/mighty-mark/` on VPS.
|
|
76
|
+
- Run `mighty-mark check security` to verify all 8 security checks.
|
|
77
|
+
- Use `verify-post-install-hardening.sh --json` for CI integration.
|
|
78
|
+
|
|
8
79
|
## [0.4.8] — 2026-03-06
|
|
9
80
|
|
|
10
81
|
**Platform hardening security checks + deployment reliability fixes.**
|
package/agent/AGENTS.md
CHANGED
|
@@ -1,7 +1,8 @@
|
|
|
1
1
|
# Mighty Mark — Operating Rules
|
|
2
2
|
|
|
3
3
|
## Rule 1: Morning Check Protocol
|
|
4
|
-
Every day at {{MORNING_CHECK_HOUR_CT}}:00 CT, I run all
|
|
4
|
+
Every day at {{MORNING_CHECK_HOUR_CT}}:00 CT, I run all health checks across 9 categories
|
|
5
|
+
(gateway, agents, system, api, data, memory, fleet, alerting, security).
|
|
5
6
|
I classify the result as GREEN, YELLOW, or RED. I send the report to Greg
|
|
6
7
|
via Telegram. I store the result for trend analysis. No exceptions, no skips.
|
|
7
8
|
|
package/agent/IDENTITY.md
CHANGED
|
@@ -15,4 +15,4 @@
|
|
|
15
15
|
| **Service** | mark-morning-check |
|
|
16
16
|
| **Infrastructure** | Hetzner VPS (shared with fleet) |
|
|
17
17
|
| **Persistence** | JSON file store at .mighty-mark-data/ |
|
|
18
|
-
| **Checks** |
|
|
18
|
+
| **Checks** | 72 across 9 categories (gateway, agents, system, api, data, memory, fleet, alerting, security) |
|
package/agent/SOUL.md
CHANGED
|
@@ -43,8 +43,8 @@ The fleet depends on me, and I depend on nothing.
|
|
|
43
43
|
## My Tools
|
|
44
44
|
|
|
45
45
|
- **`mark_status`** — Quick status snapshot (last check, active incidents, 7-day uptime)
|
|
46
|
-
- **`mark_health`** — Full health check, all
|
|
47
|
-
- **`mark_check`** — Run checks for a specific category (gateway, agents, system, api, data)
|
|
46
|
+
- **`mark_health`** — Full health check, all checks across 9 categories (gateway, agents, system, api, data, memory, fleet, alerting, security)
|
|
47
|
+
- **`mark_check`** — Run checks for a specific category (gateway, agents, system, api, data, memory, fleet, alerting, security)
|
|
48
48
|
- **`mark_incidents`** — List recent incidents from the last N days
|
|
49
49
|
- **`mark_report`** — Generate a formatted report (summary, detailed, or telegram)
|
|
50
50
|
- **`mark_uptime`** — Rolling uptime SLA data (7-day, 14-day, 30-day)
|
package/dist/index.d.ts
CHANGED
|
@@ -23,7 +23,7 @@ export declare class MarkCore {
|
|
|
23
23
|
constructor(rawConfig?: Record<string, unknown>);
|
|
24
24
|
/** Initialize all subsystems. Must be called before any operations. */
|
|
25
25
|
initialize(): Promise<void>;
|
|
26
|
-
/** Run a full health check (all
|
|
26
|
+
/** Run a full health check (all checks across 9 categories). */
|
|
27
27
|
runHealthCheck(ctx?: CheckContext): Promise<MorningReport>;
|
|
28
28
|
/** Run checks for a single category. */
|
|
29
29
|
runCategory(category: string, ctx?: CheckContext): Promise<CategoryResults>;
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAEH,OAAO,EAAe,KAAK,gBAAgB,EAAE,MAAM,2BAA2B,CAAC;AAU/E,OAAO,KAAK,EAAE,aAAa,EAAE,cAAc,EAAiB,eAAe,EAAgB,MAAM,mBAAmB,CAAC;AACrH,OAAO,KAAK,EAAE,QAAQ,EAAE,UAAU,EAAE,MAAM,qBAAqB,CAAC;AAChE,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,2BAA2B,CAAC;AAG9D,qBAAa,QAAQ;IACnB,QAAQ,CAAC,MAAM,EAAE,gBAAgB,CAAC;IAClC,OAAO,CAAC,QAAQ,CAAiB;IACjC,OAAO,CAAC,MAAM,CAAe;IAC7B,OAAO,CAAC,QAAQ,CAAoB;IACpC,OAAO,CAAC,KAAK,CAAa;IAC1B,OAAO,CAAC,aAAa,CAAiB;IACtC,OAAO,CAAC,WAAW,CAAe;IAClC,OAAO,CAAC,aAAa,CAAiB;IACtC,OAAO,CAAC,UAAU,CAA8B;gBAEpC,SAAS,GAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAM;IAInD,uEAAuE;IACjE,UAAU,IAAI,OAAO,CAAC,IAAI,CAAC;IAoBjC
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAEH,OAAO,EAAe,KAAK,gBAAgB,EAAE,MAAM,2BAA2B,CAAC;AAU/E,OAAO,KAAK,EAAE,aAAa,EAAE,cAAc,EAAiB,eAAe,EAAgB,MAAM,mBAAmB,CAAC;AACrH,OAAO,KAAK,EAAE,QAAQ,EAAE,UAAU,EAAE,MAAM,qBAAqB,CAAC;AAChE,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,2BAA2B,CAAC;AAG9D,qBAAa,QAAQ;IACnB,QAAQ,CAAC,MAAM,EAAE,gBAAgB,CAAC;IAClC,OAAO,CAAC,QAAQ,CAAiB;IACjC,OAAO,CAAC,MAAM,CAAe;IAC7B,OAAO,CAAC,QAAQ,CAAoB;IACpC,OAAO,CAAC,KAAK,CAAa;IAC1B,OAAO,CAAC,aAAa,CAAiB;IACtC,OAAO,CAAC,WAAW,CAAe;IAClC,OAAO,CAAC,aAAa,CAAiB;IACtC,OAAO,CAAC,UAAU,CAA8B;gBAEpC,SAAS,GAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAM;IAInD,uEAAuE;IACjE,UAAU,IAAI,OAAO,CAAC,IAAI,CAAC;IAoBjC,gEAAgE;IAC1D,cAAc,CAAC,GAAG,CAAC,EAAE,YAAY,GAAG,OAAO,CAAC,aAAa,CAAC;IAiBhE,wCAAwC;IAClC,WAAW,CAAC,QAAQ,EAAE,MAAM,EAAE,GAAG,CAAC,EAAE,YAAY,GAAG,OAAO,CAAC,eAAe,CAAC;IAIjF,iCAAiC;IAC3B,SAAS,IAAI,OAAO,CAAC,cAAc,CAAC;IAwB1C,0CAA0C;IACpC,YAAY,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC;IAIrD,2BAA2B;IACrB,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,UAAU,CAAC;IAIlD,mCAAmC;IAC7B,cAAc,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;IAerD,0CAA0C;IACpC,iBAAiB,CAAC,MAAM,EAAE,aAAa,GAAG,OAAO,CAAC,OAAO,CAAC;IAIhE,+CAA+C;IAC/C,OAAO,CAAC,OAAO;CAKhB;AAGD,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAC;AAClD,YAAY,EAAE,kBAAkB,EAAE,kBAAkB,EAAE,MAAM,oBAAoB,CAAC;AACjF,OAAO,EAAE,mBAAmB,EAAE,MAAM,oBAAoB,CAAC;AAGzD,OAAO,EAAE,aAAa,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAC;AACpE,YAAY,EAAE,eAAe,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AAC5E,OAAO,EAAE,iBAAiB,EAAE,MAAM,sBAAsB,CAAC;AACzD,YAAY,EAAE,eAAe,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AAC5E,OAAO,EAAE,cAAc,EAAE,MAAM,iBAAiB,CAAC;AACjD,YAAY,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAC7D,OAAO,EAAE,gBAAgB,EAAE,cAAc,EAAE,MAAM,uBAAuB,CAAC;AACzE,YAAY,EAAE,eAAe,EAAE,eAAe,EAAE,UAAU,EAAE,MAAM,uBAAuB,CAAC;AAG1F,YAAY,EAAE,gBAAgB,EAAE,MAAM,2BAA2B,CAAC;AAClE,YAAY,EAAE,aAAa,EAAE,UAAU,EAAE,iBAAiB,EAAE,MAAM,kBAAkB,CAAC;AACrF,YAAY,EACV,YAAY,EAAE,WAAW,EAAE,aAAa,EAAE,QAAQ,EAClD,WAAW,EAAE,aAAa,EAAE,cAAc,EAAE,eAAe,GAC5D,MAAM,mBAAmB,CAAC;AAC3B,YAAY,EAAE,QAAQ,EAAE,WAAW,EAAE,UAAU,EAAE,MAAM,qBAAqB,CAAC;AAC7E,YAAY,EAAE,YAAY,EAAE,MAAM,2BAA2B,CAAC;AAC9D,YAAY,EAAE,SAAS,EAAE,MAAM,kBAAkB,CAAC"}
|
package/dist/index.js
CHANGED
|
@@ -45,7 +45,7 @@ export class MarkCore {
|
|
|
45
45
|
this.trendAnalyzer = new TrendAnalyzer();
|
|
46
46
|
this.lastReport = await this.store.getLastCheck();
|
|
47
47
|
}
|
|
48
|
-
/** Run a full health check (all
|
|
48
|
+
/** Run a full health check (all checks across 9 categories). */
|
|
49
49
|
async runHealthCheck(ctx) {
|
|
50
50
|
const report = await this.runner.runAll(ctx);
|
|
51
51
|
// Persist results
|
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAEH,OAAO,EAAE,WAAW,EAAyB,MAAM,2BAA2B,CAAC;AAC/E,OAAO,EAAE,iBAAiB,EAAsB,MAAM,4BAA4B,CAAC;AACnF,OAAO,EAAE,WAAW,EAAE,MAAM,0BAA0B,CAAC;AACvD,OAAO,EAAE,gBAAgB,EAAE,MAAM,+BAA+B,CAAC;AACjE,OAAO,EAAE,mBAAmB,EAAE,oBAAoB,EAAE,oBAAoB,EAAE,MAAM,8BAA8B,CAAC;AAC/G,OAAO,EAAE,SAAS,EAAE,MAAM,uBAAuB,CAAC;AAClD,OAAO,EAAE,aAAa,EAAE,MAAM,6BAA6B,CAAC;AAC5D,OAAO,EAAE,WAAW,EAAE,MAAM,2BAA2B,CAAC;AACxD,OAAO,EAAE,aAAa,EAAE,MAAM,6BAA6B,CAAC;AAC5D,OAAO,EAAE,wBAAwB,EAAE,MAAM,sBAAsB,CAAC;AAMhE,MAAM,OAAO,QAAQ;IACV,MAAM,CAAmB;IAC1B,QAAQ,CAAiB;IACzB,MAAM,CAAe;IACrB,QAAQ,CAAoB;IAC5B,KAAK,CAAa;IAClB,aAAa,CAAiB;IAC9B,WAAW,CAAe;IAC1B,aAAa,CAAiB;IAC9B,UAAU,GAAyB,IAAI,CAAC;IAEhD,YAAY,YAAqC,EAAE;QACjD,IAAI,CAAC,MAAM,GAAG,WAAW,CAAC,SAAS,CAAC,CAAC;IACvC,CAAC;IAED,uEAAuE;IACvE,KAAK,CAAC,UAAU;QACd,IAAI,CAAC,KAAK,GAAG,IAAI,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QAChD,MAAM,IAAI,CAAC,KAAK,CAAC,UAAU,EAAE,CAAC;QAE9B,IAAI,CAAC,QAAQ,GAAG,iBAAiB,CAAC,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QACvD,IAAI,CAAC,MAAM,GAAG,IAAI,WAAW,CAAC,IAAI,CAAC,MAAM,EAAE,IAAI,CAAC,QAAQ,CAAC,CAAC;QAE1D,IAAI,CAAC,QAAQ,GAAG,IAAI,gBAAgB,CAAC;YACnC,KAAK,EAAE,IAAI,CAAC,MAAM,CAAC,aAAa,IAAI,EAAE;YACtC,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,cAAc,IAAI,EAAE;YACxC,OAAO,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,aAAa,IAAI,IAAI,CAAC,MAAM,CAAC,cAAc,CAAC;SACrE,CAAC,CAAC;QAEH,IAAI,CAAC,aAAa,GAAG,IAAI,aAAa,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QACnD,IAAI,CAAC,WAAW,GAAG,IAAI,WAAW,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QAC/C,IAAI,CAAC,aAAa,GAAG,IAAI,aAAa,EAAE,CAAC;QAEzC,IAAI,CAAC,UAAU,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,YAAY,EAAE,CAAC;IACpD,CAAC;IAED
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAEH,OAAO,EAAE,WAAW,EAAyB,MAAM,2BAA2B,CAAC;AAC/E,OAAO,EAAE,iBAAiB,EAAsB,MAAM,4BAA4B,CAAC;AACnF,OAAO,EAAE,WAAW,EAAE,MAAM,0BAA0B,CAAC;AACvD,OAAO,EAAE,gBAAgB,EAAE,MAAM,+BAA+B,CAAC;AACjE,OAAO,EAAE,mBAAmB,EAAE,oBAAoB,EAAE,oBAAoB,EAAE,MAAM,8BAA8B,CAAC;AAC/G,OAAO,EAAE,SAAS,EAAE,MAAM,uBAAuB,CAAC;AAClD,OAAO,EAAE,aAAa,EAAE,MAAM,6BAA6B,CAAC;AAC5D,OAAO,EAAE,WAAW,EAAE,MAAM,2BAA2B,CAAC;AACxD,OAAO,EAAE,aAAa,EAAE,MAAM,6BAA6B,CAAC;AAC5D,OAAO,EAAE,wBAAwB,EAAE,MAAM,sBAAsB,CAAC;AAMhE,MAAM,OAAO,QAAQ;IACV,MAAM,CAAmB;IAC1B,QAAQ,CAAiB;IACzB,MAAM,CAAe;IACrB,QAAQ,CAAoB;IAC5B,KAAK,CAAa;IAClB,aAAa,CAAiB;IAC9B,WAAW,CAAe;IAC1B,aAAa,CAAiB;IAC9B,UAAU,GAAyB,IAAI,CAAC;IAEhD,YAAY,YAAqC,EAAE;QACjD,IAAI,CAAC,MAAM,GAAG,WAAW,CAAC,SAAS,CAAC,CAAC;IACvC,CAAC;IAED,uEAAuE;IACvE,KAAK,CAAC,UAAU;QACd,IAAI,CAAC,KAAK,GAAG,IAAI,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QAChD,MAAM,IAAI,CAAC,KAAK,CAAC,UAAU,EAAE,CAAC;QAE9B,IAAI,CAAC,QAAQ,GAAG,iBAAiB,CAAC,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QACvD,IAAI,CAAC,MAAM,GAAG,IAAI,WAAW,CAAC,IAAI,CAAC,MAAM,EAAE,IAAI,CAAC,QAAQ,CAAC,CAAC;QAE1D,IAAI,CAAC,QAAQ,GAAG,IAAI,gBAAgB,CAAC;YACnC,KAAK,EAAE,IAAI,CAAC,MAAM,CAAC,aAAa,IAAI,EAAE;YACtC,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,cAAc,IAAI,EAAE;YACxC,OAAO,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,aAAa,IAAI,IAAI,CAAC,MAAM,CAAC,cAAc,CAAC;SACrE,CAAC,CAAC;QAEH,IAAI,CAAC,aAAa,GAAG,IAAI,aAAa,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QACnD,IAAI,CAAC,WAAW,GAAG,IAAI,WAAW,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QAC/C,IAAI,CAAC,aAAa,GAAG,IAAI,aAAa,EAAE,CAAC;QAEzC,IAAI,CAAC,UAAU,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,YAAY,EAAE,CAAC;IACpD,CAAC;IAED,gEAAgE;IAChE,KAAK,CAAC,cAAc,CAAC,GAAkB;QACrC,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QAE7C,kBAAkB;QAClB,MAAM,IAAI,CAAC,KAAK,CAAC,aAAa,CAAC,MAAM,CAAC,CAAC;QACvC,MAAM,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC;QAC7C,IAAI,CAAC,UAAU,GAAG,MAAM,CAAC;QAEzB,gCAAgC;QAChC,MAAM,QAAQ,GAAG,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,MAAM,KAAK,MAAM,CAAC,CAAC;QAChE,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YACxB,MAAM,IAAI,CAAC,WAAW,CAAC,kBAAkB,CAAC,QAAQ,CAAC,CAAC;QACtD,CAAC;QAED,OAAO,MAAM,CAAC;IAChB,CAAC;IAED,wCAAwC;IACxC,KAAK,CAAC,WAAW,CAAC,QAAgB,EAAE,GAAkB;QACpD,OAAO,IAAI,CAAC,MAAM,CAAC,WAAW,CAAC,QAAyB,EAAE,GAAG,CAAC,CAAC;IACjE,CAAC;IAED,iCAAiC;IACjC,KAAK,CAAC,SAAS;QACb,MAAM,eAAe,GAAG,MAAM,IAAI,CAAC,WAAW,CAAC,WAAW,EAAE,CAAC;QAC7D,IAAI,aAAa,GAAkB,IAAI,CAAC;QACxC,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC;YACrD,aAAa,GAAG,MAAM,CAAC,UAAU,CAAC;QACpC,CAAC;QAAC,MAAM,CAAC,CAAC,oBAAoB,CAAC,CAAC;QAEhC,IAAI,QAAQ,GAAG,SAAS,CAAC;QACzB,IAAI,CAAC;YACH,MAAM,EAAE,GAAG,MAAM,MAAM,CAAC,SAAS,CAAC,CAAC;YACnC,QAAQ,GAAG,EAAE,CAAC,QAAQ,EAAE,CAAC;QAC3B,CAAC;QAAC,MAAM,CAAC,CAAC,cAAc,CAAC,CAAC;QAE1B,OAAO;YACL,MAAM,EAAE,IAAI,CAAC,UAAU,EAAE,MAAM,IAAI,OAAO;YAC1C,SAAS,EAAE,IAAI,CAAC,UAAU,EAAE,SAAS,IAAI,IAAI;YAC7C,eAAe,EAAE,IAAI,CAAC,UAAU,EAAE,MAAM,IAAI,IAAI;YAChD,eAAe;YACf,eAAe,EAAE,aAAa;YAC9B,QAAQ;SACT,CAAC;IACJ,CAAC;IAED,0CAA0C;IAC1C,KAAK,CAAC,YAAY,CAAC,IAAY;QAC7B,OAAO,IAAI,CAAC,WAAW,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;IAC1C,CAAC;IAED,2BAA2B;IAC3B,KAAK,CAAC,SAAS,CAAC,IAAY;QAC1B,OAAO,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;IAC5C,CAAC;IAED,mCAAmC;IACnC,KAAK,CAAC,cAAc,CAAC,MAAc;QACjC,sDAAsD;QACtD,MAAM,MAAM,GAAG,IAAI,CAAC,OAAO,EAAE;YAC3B,CAAC,CAAC,IAAI,CAAC,UAAW;YAClB,CAAC,CAAC,MAAM,IAAI,CAAC,cAAc,EAAE,CAAC;QAEhC,QAAQ,MAAM,EAAE,CAAC;YACf,KAAK,UAAU,CAAC,CAAC,OAAO,oBAAoB,CAAC,MAAM,CAAC,CAAC;YACrD,KAAK,UAAU,CAAC,CAAC,OAAO,oBAAoB,CAAC,MAAM,CAAC,CAAC;YACrD,KAAK,SAAS,CAAC;YACf;gBACE,OAAO,mBAAmB,CAAC,MAAM,CAAC,CAAC;QACvC,CAAC;IACH,CAAC;IAED,0CAA0C;IAC1C,KAAK,CAAC,iBAAiB,CAAC,MAAqB;QAC3C,OAAO,IAAI,CAAC,QAAQ,CAAC,iBAAiB,CAAC,MAAM,CAAC,CAAC;IACjD,CAAC;IAED,+CAA+C;IACvC,OAAO;QACb,IAAI,CAAC,IAAI,CAAC,UAAU;YAAE,OAAO,KAAK,CAAC;QACnC,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC,OAAO,EAAE,CAAC;QACvE,OAAO,GAAG,GAAG,wBAAwB,CAAC;IACxC,CAAC;CACF;AAED,+DAA+D;AAC/D,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAC;AAElD,OAAO,EAAE,mBAAmB,EAAE,MAAM,oBAAoB,CAAC;AAEzD,8BAA8B;AAC9B,OAAO,EAAE,aAAa,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAC;AAEpE,OAAO,EAAE,iBAAiB,EAAE,MAAM,sBAAsB,CAAC;AAEzD,OAAO,EAAE,cAAc,EAAE,MAAM,iBAAiB,CAAC;AAEjD,OAAO,EAAE,gBAAgB,EAAE,cAAc,EAAE,MAAM,uBAAuB,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@aiassesstech/mighty-mark",
|
|
3
|
-
"version": "0.5.
|
|
3
|
+
"version": "0.5.5",
|
|
4
4
|
"description": "System Health Sentinel for AI Assess Tech Fleet — autonomous monitoring, watchdog recovery, and fleet infrastructure oversight.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -0,0 +1,433 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
set -euo pipefail
|
|
3
|
+
|
|
4
|
+
##############################################################################
|
|
5
|
+
# Fleet-Bus Diagnostics — Command-Line Health & Connectivity Checks
|
|
6
|
+
#
|
|
7
|
+
# Run from the VPS:
|
|
8
|
+
# bash /opt/mighty-mark/fleet-bus-diagnostics.sh
|
|
9
|
+
# bash /opt/mighty-mark/fleet-bus-diagnostics.sh --live # includes live ping test
|
|
10
|
+
# bash /opt/mighty-mark/fleet-bus-diagnostics.sh --verbose # show audit event details
|
|
11
|
+
#
|
|
12
|
+
# What it checks:
|
|
13
|
+
# 1. Agent cards on disk (structure, staleness)
|
|
14
|
+
# 2. Audit trail integrity (SHA-256 hash chain per agent)
|
|
15
|
+
# 3. Recent message flow (who sent what to whom)
|
|
16
|
+
# 4. Gateway fleet-bus initialization logs
|
|
17
|
+
# 5. (--live) Send a fleet/ping via gateway and check for response
|
|
18
|
+
##############################################################################
|
|
19
|
+
|
|
20
|
+
OPENCLAW_HOME="${OPENCLAW_HOME:-$HOME/.openclaw}"
|
|
21
|
+
FLEET_BUS_DIR="$OPENCLAW_HOME/workspace/fleet-bus"
|
|
22
|
+
CARDS_DIR="$FLEET_BUS_DIR/cards"
|
|
23
|
+
AUDIT_DIR="$FLEET_BUS_DIR/audit"
|
|
24
|
+
AGENTS=("jessie" "grillo" "noah" "nole" "sam" "mighty-mark")
|
|
25
|
+
LIVE_TEST=false
|
|
26
|
+
VERBOSE=false
|
|
27
|
+
|
|
28
|
+
for arg in "$@"; do
|
|
29
|
+
case "$arg" in
|
|
30
|
+
--live) LIVE_TEST=true ;;
|
|
31
|
+
--verbose) VERBOSE=true ;;
|
|
32
|
+
--help|-h)
|
|
33
|
+
echo "Usage: $0 [--live] [--verbose]"
|
|
34
|
+
echo " --live Send a live fleet/ping through the gateway"
|
|
35
|
+
echo " --verbose Show detailed audit event data"
|
|
36
|
+
exit 0
|
|
37
|
+
;;
|
|
38
|
+
esac
|
|
39
|
+
done
|
|
40
|
+
|
|
41
|
+
PASS=0
|
|
42
|
+
WARN=0
|
|
43
|
+
FAIL=0
|
|
44
|
+
|
|
45
|
+
check() {
|
|
46
|
+
local label=$1 status=$2 detail=$3
|
|
47
|
+
case "$status" in
|
|
48
|
+
PASS) echo " ✅ $label — $detail"; PASS=$((PASS + 1)) ;;
|
|
49
|
+
WARN) echo " ⚠️ $label — $detail"; WARN=$((WARN + 1)) ;;
|
|
50
|
+
FAIL) echo " ❌ $label — $detail"; FAIL=$((FAIL + 1)) ;;
|
|
51
|
+
esac
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
echo ""
|
|
55
|
+
echo "╔══════════════════════════════════════════════╗"
|
|
56
|
+
echo "║ Fleet-Bus Diagnostics ║"
|
|
57
|
+
echo "╚══════════════════════════════════════════════╝"
|
|
58
|
+
echo ""
|
|
59
|
+
echo " Fleet-bus dir: $FLEET_BUS_DIR"
|
|
60
|
+
echo " Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
|
61
|
+
echo ""
|
|
62
|
+
|
|
63
|
+
###############################################################################
|
|
64
|
+
# 1. Fleet-Bus Directory Structure
|
|
65
|
+
###############################################################################
|
|
66
|
+
echo "=== 1. Directory Structure ==="
|
|
67
|
+
|
|
68
|
+
if [ -d "$FLEET_BUS_DIR" ]; then
|
|
69
|
+
check "Fleet-bus directory" "PASS" "$FLEET_BUS_DIR exists"
|
|
70
|
+
else
|
|
71
|
+
check "Fleet-bus directory" "FAIL" "$FLEET_BUS_DIR not found"
|
|
72
|
+
echo ""
|
|
73
|
+
echo " Fleet-bus directory missing. Is the gateway running?"
|
|
74
|
+
echo " Try: systemctl restart openclaw-gateway"
|
|
75
|
+
exit 1
|
|
76
|
+
fi
|
|
77
|
+
|
|
78
|
+
if [ -d "$CARDS_DIR" ]; then
|
|
79
|
+
CARD_COUNT=$(ls -1 "$CARDS_DIR"/*.json 2>/dev/null | wc -l | tr -d ' ')
|
|
80
|
+
check "Cards directory" "PASS" "$CARDS_DIR — $CARD_COUNT card file(s)"
|
|
81
|
+
else
|
|
82
|
+
check "Cards directory" "FAIL" "$CARDS_DIR not found"
|
|
83
|
+
fi
|
|
84
|
+
|
|
85
|
+
if [ -d "$AUDIT_DIR" ]; then
|
|
86
|
+
AUDIT_COUNT=$(ls -1 "$AUDIT_DIR"/*.audit.jsonl 2>/dev/null | wc -l | tr -d ' ')
|
|
87
|
+
check "Audit directory" "PASS" "$AUDIT_DIR — $AUDIT_COUNT audit file(s)"
|
|
88
|
+
else
|
|
89
|
+
check "Audit directory" "WARN" "$AUDIT_DIR not found — no audit trail yet"
|
|
90
|
+
fi
|
|
91
|
+
echo ""
|
|
92
|
+
|
|
93
|
+
###############################################################################
|
|
94
|
+
# 2. Agent Cards
|
|
95
|
+
###############################################################################
|
|
96
|
+
echo "=== 2. Agent Cards ==="
|
|
97
|
+
|
|
98
|
+
NOW_EPOCH=$(date +%s)
|
|
99
|
+
|
|
100
|
+
for AGENT in "${AGENTS[@]}"; do
|
|
101
|
+
CARD_FILE="$CARDS_DIR/${AGENT}.json"
|
|
102
|
+
if [ ! -f "$CARD_FILE" ]; then
|
|
103
|
+
check "Card: $AGENT" "FAIL" "not found at $CARD_FILE"
|
|
104
|
+
continue
|
|
105
|
+
fi
|
|
106
|
+
|
|
107
|
+
CARD_DATA=$(python3 -c "
|
|
108
|
+
import json, sys
|
|
109
|
+
with open('$CARD_FILE') as f:
|
|
110
|
+
c = json.load(f)
|
|
111
|
+
role = c.get('role', '?')
|
|
112
|
+
status = c.get('status', '?')
|
|
113
|
+
auth = c.get('authLevel', '?')
|
|
114
|
+
mode = c.get('communicationMode', '?')
|
|
115
|
+
last_seen = c.get('lastSeen', '')
|
|
116
|
+
methods = len(c.get('supportedMethods', []))
|
|
117
|
+
print(f'{role}|{status}|{auth}|{mode}|{last_seen}|{methods}')
|
|
118
|
+
" 2>/dev/null || echo "ERROR")
|
|
119
|
+
|
|
120
|
+
if [ "$CARD_DATA" = "ERROR" ]; then
|
|
121
|
+
check "Card: $AGENT" "FAIL" "could not parse JSON"
|
|
122
|
+
continue
|
|
123
|
+
fi
|
|
124
|
+
|
|
125
|
+
IFS='|' read -r ROLE STATUS AUTH MODE LAST_SEEN METHODS <<< "$CARD_DATA"
|
|
126
|
+
|
|
127
|
+
STALE=""
|
|
128
|
+
if [ -n "$LAST_SEEN" ]; then
|
|
129
|
+
SEEN_EPOCH=$(date -d "$LAST_SEEN" +%s 2>/dev/null || date -j -f "%Y-%m-%dT%H:%M:%S" "${LAST_SEEN%%.*}" +%s 2>/dev/null || echo "0")
|
|
130
|
+
if [ "$SEEN_EPOCH" -gt 0 ]; then
|
|
131
|
+
AGE_HOURS=$(( (NOW_EPOCH - SEEN_EPOCH) / 3600 ))
|
|
132
|
+
if [ "$AGE_HOURS" -gt 48 ]; then
|
|
133
|
+
STALE=" ⚠️ stale (${AGE_HOURS}h ago)"
|
|
134
|
+
fi
|
|
135
|
+
fi
|
|
136
|
+
fi
|
|
137
|
+
|
|
138
|
+
check "Card: $AGENT" "PASS" "role=$ROLE status=$STATUS auth=$AUTH mode=$MODE methods=$METHODS${STALE}"
|
|
139
|
+
done
|
|
140
|
+
echo ""
|
|
141
|
+
|
|
142
|
+
###############################################################################
|
|
143
|
+
# 3. Audit Trail Integrity
|
|
144
|
+
###############################################################################
|
|
145
|
+
echo "=== 3. Audit Trail ==="
|
|
146
|
+
|
|
147
|
+
TOTAL_EVENTS=0
|
|
148
|
+
declare -A AGENT_EVENTS
|
|
149
|
+
|
|
150
|
+
for AGENT in "${AGENTS[@]}"; do
|
|
151
|
+
AUDIT_FILE="$AUDIT_DIR/${AGENT}.audit.jsonl"
|
|
152
|
+
if [ ! -f "$AUDIT_FILE" ]; then
|
|
153
|
+
check "Audit: $AGENT" "WARN" "no audit file yet"
|
|
154
|
+
AGENT_EVENTS[$AGENT]=0
|
|
155
|
+
continue
|
|
156
|
+
fi
|
|
157
|
+
|
|
158
|
+
EVENT_COUNT=$(wc -l < "$AUDIT_FILE" | tr -d ' ')
|
|
159
|
+
TOTAL_EVENTS=$((TOTAL_EVENTS + EVENT_COUNT))
|
|
160
|
+
AGENT_EVENTS[$AGENT]=$EVENT_COUNT
|
|
161
|
+
|
|
162
|
+
CHAIN_FILE="$AUDIT_DIR/${AGENT}.chain.json"
|
|
163
|
+
if [ -f "$CHAIN_FILE" ]; then
|
|
164
|
+
CHAIN_COUNT=$(python3 -c "
|
|
165
|
+
import json
|
|
166
|
+
with open('$CHAIN_FILE') as f:
|
|
167
|
+
c = json.load(f)
|
|
168
|
+
print(c.get('eventCount', 0))
|
|
169
|
+
" 2>/dev/null || echo "?")
|
|
170
|
+
if [ "$CHAIN_COUNT" = "$EVENT_COUNT" ]; then
|
|
171
|
+
check "Audit: $AGENT" "PASS" "$EVENT_COUNT events, chain state consistent"
|
|
172
|
+
else
|
|
173
|
+
check "Audit: $AGENT" "WARN" "$EVENT_COUNT events in JSONL but chain says $CHAIN_COUNT"
|
|
174
|
+
fi
|
|
175
|
+
else
|
|
176
|
+
check "Audit: $AGENT" "WARN" "$EVENT_COUNT events but no chain state file"
|
|
177
|
+
fi
|
|
178
|
+
done
|
|
179
|
+
|
|
180
|
+
echo ""
|
|
181
|
+
echo " Total audit events across fleet: $TOTAL_EVENTS"
|
|
182
|
+
echo ""
|
|
183
|
+
|
|
184
|
+
###############################################################################
|
|
185
|
+
# 4. Recent Message Flow
|
|
186
|
+
###############################################################################
|
|
187
|
+
echo "=== 4. Message Flow (Last 50 Events) ==="
|
|
188
|
+
|
|
189
|
+
HAS_MESSAGES=false
|
|
190
|
+
|
|
191
|
+
for AGENT in "${AGENTS[@]}"; do
|
|
192
|
+
AUDIT_FILE="$AUDIT_DIR/${AGENT}.audit.jsonl"
|
|
193
|
+
[ ! -f "$AUDIT_FILE" ] && continue
|
|
194
|
+
[ "${AGENT_EVENTS[$AGENT]}" = "0" ] && continue
|
|
195
|
+
|
|
196
|
+
FLOW=$(python3 -c "
|
|
197
|
+
import json, sys
|
|
198
|
+
|
|
199
|
+
events = []
|
|
200
|
+
with open('$AUDIT_FILE') as f:
|
|
201
|
+
for line in f:
|
|
202
|
+
line = line.strip()
|
|
203
|
+
if not line:
|
|
204
|
+
continue
|
|
205
|
+
try:
|
|
206
|
+
events.append(json.loads(line))
|
|
207
|
+
except:
|
|
208
|
+
pass
|
|
209
|
+
|
|
210
|
+
recent = events[-50:] if len(events) > 50 else events
|
|
211
|
+
|
|
212
|
+
sent = []
|
|
213
|
+
received = []
|
|
214
|
+
violations = []
|
|
215
|
+
errors = []
|
|
216
|
+
|
|
217
|
+
for e in recent:
|
|
218
|
+
et = e.get('eventType', '')
|
|
219
|
+
msg = e.get('message', {}) or {}
|
|
220
|
+
ts = e.get('timestamp', '?')[:19]
|
|
221
|
+
outcome = e.get('outcome', '?')
|
|
222
|
+
fr = msg.get('from', '?')
|
|
223
|
+
to = msg.get('to', '?')
|
|
224
|
+
method = msg.get('method', '?')
|
|
225
|
+
|
|
226
|
+
if et == 'message_sent':
|
|
227
|
+
sent.append(f' {ts} {fr} → {to} [{method}] outcome={outcome}')
|
|
228
|
+
elif et == 'message_received':
|
|
229
|
+
received.append(f' {ts} {fr} → {to} [{method}] outcome={outcome}')
|
|
230
|
+
elif et == 'routing_violation':
|
|
231
|
+
violations.append(f' {ts} {fr} → {to} [{method}] reason={e.get(\"routingDecision\", \"?\")}'[:120])
|
|
232
|
+
elif outcome in ('error', 'failed'):
|
|
233
|
+
errors.append(f' {ts} {et} outcome={outcome} details={e.get(\"details\", \"?\")}'[:120])
|
|
234
|
+
|
|
235
|
+
verbose = '$VERBOSE' == 'true'
|
|
236
|
+
if sent:
|
|
237
|
+
print(f' 📤 Sent ({len(sent)}):')
|
|
238
|
+
for s in (sent if verbose else sent[-5:]):
|
|
239
|
+
print(s)
|
|
240
|
+
if not verbose and len(sent) > 5:
|
|
241
|
+
print(f' ... and {len(sent) - 5} more (use --verbose)')
|
|
242
|
+
if received:
|
|
243
|
+
print(f' 📥 Received ({len(received)}):')
|
|
244
|
+
for r in (received if verbose else received[-5:]):
|
|
245
|
+
print(r)
|
|
246
|
+
if not verbose and len(received) > 5:
|
|
247
|
+
print(f' ... and {len(received) - 5} more (use --verbose)')
|
|
248
|
+
if violations:
|
|
249
|
+
print(f' 🚫 Routing Violations ({len(violations)}):')
|
|
250
|
+
for v in violations:
|
|
251
|
+
print(v)
|
|
252
|
+
if errors:
|
|
253
|
+
print(f' 💥 Errors ({len(errors)}):')
|
|
254
|
+
for e in errors:
|
|
255
|
+
print(e)
|
|
256
|
+
|
|
257
|
+
if sent or received:
|
|
258
|
+
sys.exit(0)
|
|
259
|
+
elif violations or errors:
|
|
260
|
+
sys.exit(2)
|
|
261
|
+
else:
|
|
262
|
+
sys.exit(1)
|
|
263
|
+
" 2>/dev/null)
|
|
264
|
+
|
|
265
|
+
EXIT_CODE=$?
|
|
266
|
+
|
|
267
|
+
if [ $EXIT_CODE -eq 0 ]; then
|
|
268
|
+
HAS_MESSAGES=true
|
|
269
|
+
echo " ── $AGENT ──"
|
|
270
|
+
echo "$FLOW"
|
|
271
|
+
echo ""
|
|
272
|
+
elif [ $EXIT_CODE -eq 2 ]; then
|
|
273
|
+
HAS_MESSAGES=true
|
|
274
|
+
echo " ── $AGENT (issues detected) ──"
|
|
275
|
+
echo "$FLOW"
|
|
276
|
+
echo ""
|
|
277
|
+
fi
|
|
278
|
+
done
|
|
279
|
+
|
|
280
|
+
if ! $HAS_MESSAGES; then
|
|
281
|
+
echo " No fleet messages found in any agent's audit trail."
|
|
282
|
+
echo " This means agents haven't communicated via fleet-bus yet."
|
|
283
|
+
echo ""
|
|
284
|
+
echo " To trigger a test, ask Jessie to delegate a task to Sam:"
|
|
285
|
+
echo ' "Jessie, ask Sam to write a hello world script"'
|
|
286
|
+
echo ""
|
|
287
|
+
fi
|
|
288
|
+
|
|
289
|
+
###############################################################################
|
|
290
|
+
# 5. Gateway Fleet-Bus Initialization
|
|
291
|
+
###############################################################################
|
|
292
|
+
echo "=== 5. Gateway Initialization ==="
|
|
293
|
+
|
|
294
|
+
if command -v journalctl &>/dev/null; then
|
|
295
|
+
INIT_LOGS=$(journalctl -u openclaw-gateway --since "24 hours ago" --no-pager 2>/dev/null | grep -i 'fleet-bus.*initialized' || true)
|
|
296
|
+
|
|
297
|
+
if [ -n "$INIT_LOGS" ]; then
|
|
298
|
+
INIT_COUNT=$(echo "$INIT_LOGS" | wc -l | tr -d ' ')
|
|
299
|
+
check "Fleet-bus init logs" "PASS" "$INIT_COUNT initialization(s) in last 24h"
|
|
300
|
+
|
|
301
|
+
INITIALIZED_AGENTS=""
|
|
302
|
+
for AGENT in "${AGENTS[@]}"; do
|
|
303
|
+
if echo "$INIT_LOGS" | grep -q "fleet-bus:${AGENT}"; then
|
|
304
|
+
INITIALIZED_AGENTS="$INITIALIZED_AGENTS $AGENT"
|
|
305
|
+
fi
|
|
306
|
+
done
|
|
307
|
+
|
|
308
|
+
MISSING_AGENTS=""
|
|
309
|
+
for AGENT in "${AGENTS[@]}"; do
|
|
310
|
+
if ! echo "$INITIALIZED_AGENTS" | grep -q "$AGENT"; then
|
|
311
|
+
MISSING_AGENTS="$MISSING_AGENTS $AGENT"
|
|
312
|
+
fi
|
|
313
|
+
done
|
|
314
|
+
|
|
315
|
+
if [ -n "$MISSING_AGENTS" ]; then
|
|
316
|
+
check "Agent init coverage" "WARN" "not seen in logs:$MISSING_AGENTS"
|
|
317
|
+
else
|
|
318
|
+
check "Agent init coverage" "PASS" "all 6 agents initialized"
|
|
319
|
+
fi
|
|
320
|
+
|
|
321
|
+
if $VERBOSE; then
|
|
322
|
+
echo ""
|
|
323
|
+
echo " Recent init logs:"
|
|
324
|
+
echo "$INIT_LOGS" | tail -12 | while read -r line; do
|
|
325
|
+
echo " $line"
|
|
326
|
+
done
|
|
327
|
+
fi
|
|
328
|
+
else
|
|
329
|
+
check "Fleet-bus init logs" "WARN" "no fleet-bus initialization in last 24h"
|
|
330
|
+
fi
|
|
331
|
+
else
|
|
332
|
+
check "Fleet-bus init logs" "WARN" "journalctl not available — cannot check gateway logs"
|
|
333
|
+
fi
|
|
334
|
+
echo ""
|
|
335
|
+
|
|
336
|
+
###############################################################################
|
|
337
|
+
# 6. Fleet-Bus Transport Test (requires --live)
|
|
338
|
+
###############################################################################
|
|
339
|
+
if $LIVE_TEST; then
|
|
340
|
+
echo "=== 6. Live Transport Test (fleet/ping) ==="
|
|
341
|
+
|
|
342
|
+
GW_PORT=$(python3 -c "
|
|
343
|
+
import json
|
|
344
|
+
with open('$OPENCLAW_HOME/openclaw.json') as f:
|
|
345
|
+
cfg = json.load(f)
|
|
346
|
+
print(cfg.get('gateway', {}).get('port', 18789))
|
|
347
|
+
" 2>/dev/null || echo "18789")
|
|
348
|
+
|
|
349
|
+
GW_TOKEN=$(python3 -c "
|
|
350
|
+
import json
|
|
351
|
+
with open('$OPENCLAW_HOME/openclaw.json') as f:
|
|
352
|
+
cfg = json.load(f)
|
|
353
|
+
print(cfg.get('gateway', {}).get('auth', {}).get('token', ''))
|
|
354
|
+
" 2>/dev/null || echo "")
|
|
355
|
+
|
|
356
|
+
GW_URL="http://127.0.0.1:${GW_PORT}"
|
|
357
|
+
|
|
358
|
+
if [ -z "$GW_TOKEN" ]; then
|
|
359
|
+
check "Gateway token" "WARN" "no auth token found — live test may fail"
|
|
360
|
+
fi
|
|
361
|
+
|
|
362
|
+
echo " Gateway: $GW_URL"
|
|
363
|
+
echo ""
|
|
364
|
+
|
|
365
|
+
PING_MSG=$(python3 -c "
|
|
366
|
+
import json, time, secrets
|
|
367
|
+
msg = {
|
|
368
|
+
'id': f'fm-diag-{int(time.time())}-{secrets.token_hex(4)}',
|
|
369
|
+
'from': 'mighty-mark',
|
|
370
|
+
'to': 'jessie',
|
|
371
|
+
'method': 'fleet/ping',
|
|
372
|
+
'params': {'source': 'diagnostics', 'timestamp': '$(date -u +%Y-%m-%dT%H:%M:%SZ)'},
|
|
373
|
+
'timestamp': '$(date -u +%Y-%m-%dT%H:%M:%SZ)',
|
|
374
|
+
'nonce': secrets.token_hex(8),
|
|
375
|
+
'version': '1.0'
|
|
376
|
+
}
|
|
377
|
+
print('%%FLEET%%' + json.dumps(msg))
|
|
378
|
+
")
|
|
379
|
+
|
|
380
|
+
RESPONSE=$(curl -s -w "\n%{http_code}" \
|
|
381
|
+
-X POST "$GW_URL/api/v1/agent" \
|
|
382
|
+
-H "Content-Type: application/json" \
|
|
383
|
+
-H "Authorization: Bearer $GW_TOKEN" \
|
|
384
|
+
-d "$(python3 -c "
|
|
385
|
+
import json
|
|
386
|
+
print(json.dumps({
|
|
387
|
+
'agentId': 'jessie',
|
|
388
|
+
'message': '''$PING_MSG''',
|
|
389
|
+
'deliver': False,
|
|
390
|
+
'timeoutSeconds': 10,
|
|
391
|
+
'label': 'fleet:fleet/ping'
|
|
392
|
+
}))
|
|
393
|
+
")" 2>/dev/null || echo -e "\n000")
|
|
394
|
+
|
|
395
|
+
HTTP_CODE=$(echo "$RESPONSE" | tail -1)
|
|
396
|
+
BODY=$(echo "$RESPONSE" | head -n -1)
|
|
397
|
+
|
|
398
|
+
if [ "$HTTP_CODE" = "200" ]; then
|
|
399
|
+
check "Live ping (Mark → Jessie)" "PASS" "HTTP 200 — gateway accepted fleet/ping"
|
|
400
|
+
if $VERBOSE; then
|
|
401
|
+
echo " Response: $(echo "$BODY" | head -c 200)"
|
|
402
|
+
fi
|
|
403
|
+
elif [ "$HTTP_CODE" = "401" ] || [ "$HTTP_CODE" = "403" ]; then
|
|
404
|
+
check "Live ping (Mark → Jessie)" "FAIL" "HTTP $HTTP_CODE — auth rejected (check gateway token)"
|
|
405
|
+
elif [ "$HTTP_CODE" = "000" ]; then
|
|
406
|
+
check "Live ping (Mark → Jessie)" "FAIL" "connection refused — is the gateway running?"
|
|
407
|
+
else
|
|
408
|
+
check "Live ping (Mark → Jessie)" "WARN" "HTTP $HTTP_CODE — unexpected response"
|
|
409
|
+
if $VERBOSE; then
|
|
410
|
+
echo " Response: $(echo "$BODY" | head -c 200)"
|
|
411
|
+
fi
|
|
412
|
+
fi
|
|
413
|
+
echo ""
|
|
414
|
+
fi
|
|
415
|
+
|
|
416
|
+
###############################################################################
|
|
417
|
+
# Summary
|
|
418
|
+
###############################################################################
|
|
419
|
+
echo "══════════════════════════════════════════════"
|
|
420
|
+
echo " Results: $PASS passed, $WARN warnings, $FAIL failed"
|
|
421
|
+
echo "══════════════════════════════════════════════"
|
|
422
|
+
echo ""
|
|
423
|
+
|
|
424
|
+
if [ $FAIL -gt 0 ]; then
|
|
425
|
+
echo " ⚡ Fleet-bus has $FAIL failing checks — investigate above."
|
|
426
|
+
exit 1
|
|
427
|
+
elif [ $WARN -gt 0 ]; then
|
|
428
|
+
echo " ⚠️ $WARN warnings — review above for details."
|
|
429
|
+
exit 0
|
|
430
|
+
else
|
|
431
|
+
echo " 🎯 Fleet-bus is healthy."
|
|
432
|
+
exit 0
|
|
433
|
+
fi
|
|
@@ -397,7 +397,7 @@ read -r -d '' MARK_MEMORY << 'MEMEOF' || true
|
|
|
397
397
|
|
|
398
398
|
## Morning Check Protocol
|
|
399
399
|
|
|
400
|
-
- Daily at 06:00 CT. 72 checks across
|
|
400
|
+
- Daily at 06:00 CT. 72 checks across 9 categories: gateway, agents, system, api, data, memory, fleet, alerting, security.
|
|
401
401
|
- Classify: GREEN (all pass), YELLOW (warnings), RED (failures). Send to Greg via Telegram.
|
|
402
402
|
- Fleet backup runs BEFORE health checks. Full (Sunday, ~460MB) and Light (Mon-Sat, ~5-15MB).
|
|
403
403
|
- Backup pushed off-site to GitHub. 35-day retention. Both tiers have test suites (48 bash tests).
|