@cohortapp/agent-sdk 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/commands/init-agent.md +104 -0
- package/.claude/commands/init-maestro.md +1187 -0
- package/.claude/settings.json +161 -0
- package/.env.example +216 -0
- package/README.md +632 -0
- package/agents/browser-operator/agent.md +52 -0
- package/agents/calendar-ops/agent.md +50 -0
- package/agents/communications/agent.md +96 -0
- package/agents/decision-log/agent.md +65 -0
- package/agents/desktop-operator/agent.md +59 -0
- package/agents/gmail-operator/agent.md +62 -0
- package/agents/inbound-dispatcher/agent.md +66 -0
- package/agents/inbox-processor/agent.md +39 -0
- package/agents/pmo-execution/agent.md +60 -0
- package/agents/session-spawner/agent.md +64 -0
- package/agents/slack-operator/agent.md +60 -0
- package/agents/whatsapp-operator/agent.md +60 -0
- package/agents/workflow-automation/agent.md +61 -0
- package/archetypes/altitudes/c-suite.yaml +58 -0
- package/archetypes/altitudes/founder.yaml +68 -0
- package/archetypes/altitudes/senior-manager.yaml +63 -0
- package/archetypes/altitudes/svp.yaml +60 -0
- package/archetypes/altitudes/vp.yaml +50 -0
- package/archetypes/archetype.schema.json +77 -0
- package/archetypes/base.yaml +47 -0
- package/archetypes/capabilities/commercial-leader.yaml +159 -0
- package/archetypes/capabilities/compliance-officer.yaml +159 -0
- package/archetypes/capabilities/executive-operator.yaml +169 -0
- package/archetypes/capabilities/finance-leader.yaml +162 -0
- package/archetypes/capabilities/operations-leader.yaml +154 -0
- package/archetypes/capabilities/product-leader.yaml +148 -0
- package/archetypes/capabilities/technical-leader.yaml +146 -0
- package/archetypes/functions/commercial-leader.yaml +62 -0
- package/archetypes/functions/compliance-officer.yaml +64 -0
- package/archetypes/functions/executive-operator.yaml +70 -0
- package/archetypes/functions/finance-leader.yaml +70 -0
- package/archetypes/functions/operations-leader.yaml +62 -0
- package/archetypes/functions/product-leader.yaml +61 -0
- package/archetypes/functions/technical-leader.yaml +57 -0
- package/bin/cohort-mcp.mjs +81 -0
- package/bin/maestro.mjs +3516 -0
- package/bin/maestro.test.mjs +1015 -0
- package/desktop-control/README.md +56 -0
- package/desktop-control/app-profiles/gmail.yaml +120 -0
- package/desktop-control/app-profiles/slack.yaml +315 -0
- package/desktop-control/app-profiles/whatsapp.yaml +107 -0
- package/docs/architecture/agent-topology.md +2239 -0
- package/docs/architecture/archetype-agent-factory.md +110 -0
- package/docs/architecture/collective-memory-and-org-mesh.md +115 -0
- package/docs/architecture/continuous-monitoring.md +221 -0
- package/docs/architecture/mcp-capability-map.md +585 -0
- package/docs/architecture/system-architecture.md +1272 -0
- package/docs/company-context/README.md +40 -0
- package/docs/guides/agent-persona-setup.md +600 -0
- package/docs/guides/agents-observe-setup.md +64 -0
- package/docs/guides/billing-console-keys.md +88 -0
- package/docs/guides/ccxray-diagnostics.md +65 -0
- package/docs/guides/channel-bus.md +127 -0
- package/docs/guides/claude-mem-setup.md +79 -0
- package/docs/guides/claude-pace-setup.md +56 -0
- package/docs/guides/claudraband-sessions.md +98 -0
- package/docs/guides/clawteam-swarm.md +116 -0
- package/docs/guides/code-review-graph-setup.md +86 -0
- package/docs/guides/email-setup.md +431 -0
- package/docs/guides/mac-mini.md +119 -0
- package/docs/guides/media-generation-setup.md +349 -0
- package/docs/guides/model-routing.md +162 -0
- package/docs/guides/observability-otel.md +265 -0
- package/docs/guides/org-onboarding.md +132 -0
- package/docs/guides/outbound-governance-setup.md +437 -0
- package/docs/guides/pdf-generation-setup.md +315 -0
- package/docs/guides/poller-daemon-setup.md +563 -0
- package/docs/guides/rag-context-setup.md +459 -0
- package/docs/guides/self-optimization-pattern.md +82 -0
- package/docs/guides/setup-wizard.md +178 -0
- package/docs/guides/slack-setup.md +350 -0
- package/docs/guides/telegram-setup.md +227 -0
- package/docs/guides/twilio-subaccounts-setup.md +223 -0
- package/docs/guides/verification.md +128 -0
- package/docs/guides/voice-mode.md +188 -0
- package/docs/guides/voice-sms-setup.md +698 -0
- package/docs/guides/webhook-relay-setup.md +349 -0
- package/docs/guides/whatsapp-setup.md +288 -0
- package/docs/prompts/board-pack-cover-template.md +36 -0
- package/docs/prompts/decision-recommendation-template.md +88 -0
- package/docs/prompts/followup-message-template.md +141 -0
- package/docs/prompts/investor-letter-template.md +52 -0
- package/docs/prompts/morning-brief-template.md +82 -0
- package/docs/prompts/presentation-template.md +58 -0
- package/docs/prompts/weekly-strategic-memo-template.md +104 -0
- package/docs/research/hallucinated-tool-output-investigation.md +151 -0
- package/docs/runbooks/backup-restore.md +205 -0
- package/docs/runbooks/cohort-cutover.md +129 -0
- package/docs/runbooks/fleet-operations.md +200 -0
- package/docs/runbooks/incident-response.md +226 -0
- package/docs/runbooks/mac-mini-bootstrap.md +431 -0
- package/docs/runbooks/perpetual-operations.md +509 -0
- package/docs/runbooks/recovery-and-failover.md +260 -0
- package/framework-features.json +267 -0
- package/ingest/README.md +87 -0
- package/lib/action-executor.js +689 -0
- package/lib/action-executor.test.mjs +871 -0
- package/lib/agent-root.mjs +37 -0
- package/lib/archetype.mjs +236 -0
- package/lib/archetype.test.mjs +132 -0
- package/lib/autonomy.mjs +114 -0
- package/lib/autonomy.test.mjs +66 -0
- package/lib/backlog.mjs +358 -0
- package/lib/backlog.test.mjs +266 -0
- package/lib/budget-guard.mjs +279 -0
- package/lib/budget-guard.test.mjs +291 -0
- package/lib/cadence-bus-schedule.test.mjs +194 -0
- package/lib/cadence-bus.mjs +1120 -0
- package/lib/cadence-bus.test.mjs +720 -0
- package/lib/cadences.mjs +205 -0
- package/lib/cadences.test.mjs +125 -0
- package/lib/capability.mjs +154 -0
- package/lib/capability.test.mjs +78 -0
- package/lib/channels/base-adapter.mjs +719 -0
- package/lib/channels/base-adapter.test.mjs +590 -0
- package/lib/channels/channel.mjs +128 -0
- package/lib/channels/channels.test.mjs +371 -0
- package/lib/channels/contract.mjs +215 -0
- package/lib/channels/contract.test.mjs +137 -0
- package/lib/channels/conversation-resolver.mjs +95 -0
- package/lib/channels/gmail/adapter.mjs +87 -0
- package/lib/channels/inbox-item.mjs +255 -0
- package/lib/channels/inbox-item.test.mjs +335 -0
- package/lib/channels/index.mjs +94 -0
- package/lib/channels/orgmail/adapter.mjs +353 -0
- package/lib/channels/orgmail/adapter.test.mjs +311 -0
- package/lib/channels/pairing.mjs +363 -0
- package/lib/channels/pairing.test.mjs +270 -0
- package/lib/channels/registry.mjs +164 -0
- package/lib/channels/slack/adapter.mjs +317 -0
- package/lib/channels/slack-adapter.test.mjs +212 -0
- package/lib/channels/sms/adapter.mjs +43 -0
- package/lib/channels/telegram/adapter.mjs +432 -0
- package/lib/channels/telegram-adapter.test.mjs +306 -0
- package/lib/channels/voice/adapter.mjs +301 -0
- package/lib/channels/voice/adapter.test.mjs +278 -0
- package/lib/channels/whatsapp/adapter-baileys.mjs +587 -0
- package/lib/channels/whatsapp/adapter-baileys.test.mjs +359 -0
- package/lib/channels/whatsapp/adapter-twilio.mjs +65 -0
- package/lib/channels/whatsapp/baileys-typing.test.mjs +154 -0
- package/lib/charter.mjs +256 -0
- package/lib/charter.test.mjs +89 -0
- package/lib/claude-bin.mjs +134 -0
- package/lib/claude-bin.test.mjs +75 -0
- package/lib/collective/capture.mjs +185 -0
- package/lib/collective/capture.test.mjs +121 -0
- package/lib/collective/cards.mjs +201 -0
- package/lib/collective/cards.test.mjs +114 -0
- package/lib/collective/config.mjs +186 -0
- package/lib/collective/config.test.mjs +123 -0
- package/lib/collective/global-config.mjs +113 -0
- package/lib/collective/global-config.test.mjs +75 -0
- package/lib/collective/presence.mjs +201 -0
- package/lib/collective/presence.test.mjs +95 -0
- package/lib/collective/recall.mjs +215 -0
- package/lib/collective/recall.test.mjs +116 -0
- package/lib/comms/send-gate.mjs +554 -0
- package/lib/comms/send-gate.test.mjs +577 -0
- package/lib/comms.mjs +67 -0
- package/lib/comms.test.mjs +41 -0
- package/lib/diagnostics/alerts.mjs +424 -0
- package/lib/diagnostics/alerts.test.mjs +318 -0
- package/lib/diagnostics/backup-freshness.mjs +188 -0
- package/lib/diagnostics/backup-freshness.test.mjs +185 -0
- package/lib/diagnostics/counters.mjs +269 -0
- package/lib/diagnostics/counters.test.mjs +206 -0
- package/lib/diagnostics/events.mjs +188 -0
- package/lib/diagnostics/events.test.mjs +290 -0
- package/lib/diagnostics/otel.mjs +237 -0
- package/lib/diagnostics/otel.test.mjs +196 -0
- package/lib/diagnostics/trace.mjs +216 -0
- package/lib/diagnostics/trace.test.mjs +251 -0
- package/lib/env-compat.mjs +74 -0
- package/lib/env-compat.test.mjs +104 -0
- package/lib/feature-init.mjs +331 -0
- package/lib/fs-atomic.mjs +112 -0
- package/lib/fs-atomic.test.mjs +72 -0
- package/lib/fs-ownership.mjs +111 -0
- package/lib/fs-ownership.test.mjs +158 -0
- package/lib/hooks/bus.mjs +347 -0
- package/lib/hooks/bus.test.mjs +387 -0
- package/lib/index.js +16 -0
- package/lib/learning/config.mjs +106 -0
- package/lib/learning/config.test.mjs +75 -0
- package/lib/learning/counters.mjs +156 -0
- package/lib/learning/counters.test.mjs +69 -0
- package/lib/learning/curator-consolidate.test.mjs +238 -0
- package/lib/learning/curator.mjs +453 -0
- package/lib/learning/curator.test.mjs +106 -0
- package/lib/learning/log.mjs +40 -0
- package/lib/learning/reflect.mjs +534 -0
- package/lib/learning/reflect.test.mjs +0 -0
- package/lib/learning/session-index.mjs +352 -0
- package/lib/learning/session-index.test.mjs +125 -0
- package/lib/learning/skill-writer.mjs +474 -0
- package/lib/learning/skill-writer.test.mjs +210 -0
- package/lib/mcp/server.mjs +328 -0
- package/lib/mcp/server.test.mjs +400 -0
- package/lib/model-router/auth-profiles.mjs +758 -0
- package/lib/model-router/auth-profiles.test.mjs +580 -0
- package/lib/model-router/catalog/anthropic.yaml +153 -0
- package/lib/model-router/catalog/deepseek.yaml +86 -0
- package/lib/model-router/catalog/moonshot.yaml +81 -0
- package/lib/model-router/catalog/qwen.yaml +114 -0
- package/lib/model-router/catalog.mjs +925 -0
- package/lib/model-router/catalog.test.mjs +385 -0
- package/lib/model-router/economics.mjs +564 -0
- package/lib/model-router/economics.test.mjs +344 -0
- package/lib/model-router/failover.mjs +298 -0
- package/lib/model-router/failover.test.mjs +439 -0
- package/lib/model-router/health.mjs +453 -0
- package/lib/model-router/health.test.mjs +338 -0
- package/lib/model-router/integration-coverage.test.mjs +829 -0
- package/lib/model-router/integration.test.mjs +564 -0
- package/lib/model-router/ledger.mjs +402 -0
- package/lib/model-router/ledger.test.mjs +382 -0
- package/lib/model-router/llm-task.mjs +515 -0
- package/lib/model-router/llm-task.test.mjs +392 -0
- package/lib/model-router/org-credentials.mjs +260 -0
- package/lib/model-router/org-credentials.test.mjs +265 -0
- package/lib/model-router/pricing-refresh.mjs +463 -0
- package/lib/model-router/pricing-refresh.test.mjs +286 -0
- package/lib/model-router/reconcile.mjs +429 -0
- package/lib/model-router/reconcile.test.mjs +316 -0
- package/lib/model-router/repair.mjs +471 -0
- package/lib/model-router/repair.test.mjs +180 -0
- package/lib/model-router/resolve.mjs +1206 -0
- package/lib/model-router/spawn.mjs +497 -0
- package/lib/model-router/spawn.test.mjs +425 -0
- package/lib/model-router/taxonomy.mjs +893 -0
- package/lib/model-router/taxonomy.test.mjs +410 -0
- package/lib/model-router.mjs +677 -0
- package/lib/model-router.test.mjs +907 -0
- package/lib/org/activity.mjs +211 -0
- package/lib/org/activity.test.mjs +134 -0
- package/lib/org/approvals.mjs +448 -0
- package/lib/org/approvals.test.mjs +216 -0
- package/lib/org/awareness.mjs +222 -0
- package/lib/org/awareness.test.mjs +159 -0
- package/lib/org/board.mjs +229 -0
- package/lib/org/board.test.mjs +177 -0
- package/lib/org/bootstrap-context.mjs +169 -0
- package/lib/org/bootstrap-context.test.mjs +153 -0
- package/lib/org/client.mjs +1628 -0
- package/lib/org/client.test.mjs +1107 -0
- package/lib/org/cohort-client.mjs +67 -0
- package/lib/org/cohort-client.test.mjs +126 -0
- package/lib/org/cost-sync.mjs +227 -0
- package/lib/org/cost-sync.test.mjs +153 -0
- package/lib/org/doctor.mjs +212 -0
- package/lib/org/doctor.test.mjs +212 -0
- package/lib/org/handoff.mjs +293 -0
- package/lib/org/handoff.test.mjs +269 -0
- package/lib/org/integration-tools.mjs +182 -0
- package/lib/org/integration-tools.test.mjs +160 -0
- package/lib/org/keys.mjs +131 -0
- package/lib/org/keys.test.mjs +92 -0
- package/lib/org/knowledge.mjs +463 -0
- package/lib/org/knowledge.test.mjs +319 -0
- package/lib/org/leases.mjs +335 -0
- package/lib/org/leases.test.mjs +235 -0
- package/lib/org/mesh-integration.test.mjs +127 -0
- package/lib/org/mesh.mjs +459 -0
- package/lib/org/mesh.test.mjs +345 -0
- package/lib/org/messaging.mjs +503 -0
- package/lib/org/messaging.test.mjs +238 -0
- package/lib/org/policy.mjs +345 -0
- package/lib/org/policy.test.mjs +237 -0
- package/lib/org/protocol.checksum +1 -0
- package/lib/org/protocol.checksum.test.mjs +90 -0
- package/lib/org/protocol.mjs +967 -0
- package/lib/org/protocol.test.mjs +264 -0
- package/lib/org/registry.mjs +194 -0
- package/lib/org/registry.test.mjs +100 -0
- package/lib/org/tool-surface-integration.test.mjs +120 -0
- package/lib/org/tool-surface.mjs +2535 -0
- package/lib/org/tool-surface.test.mjs +589 -0
- package/lib/org/ui-parity.mjs +3236 -0
- package/lib/org/ui-parity.test.mjs +348 -0
- package/lib/org/verify.mjs +176 -0
- package/lib/org/verify.test.mjs +194 -0
- package/lib/rag/embed.mjs +188 -0
- package/lib/rag/indexer.mjs +425 -0
- package/lib/rag/rag.test.mjs +505 -0
- package/lib/rag/search.mjs +475 -0
- package/lib/rate-guard.mjs +246 -0
- package/lib/rate-guard.test.mjs +201 -0
- package/lib/render.mjs +112 -0
- package/lib/render.test.mjs +68 -0
- package/lib/resource-governor.mjs +297 -0
- package/lib/resource-governor.test.mjs +262 -0
- package/lib/scheduling/dynamic-jobs.mjs +675 -0
- package/lib/scheduling/dynamic-jobs.test.mjs +344 -0
- package/lib/scheduling/jitter.mjs +0 -0
- package/lib/scheduling/jitter.test.mjs +140 -0
- package/lib/secrets/broker.mjs +315 -0
- package/lib/secrets/broker.test.mjs +280 -0
- package/lib/secrets/providers.mjs +461 -0
- package/lib/secrets/providers.test.mjs +274 -0
- package/lib/security/audit-engine.mjs +684 -0
- package/lib/security/audit-engine.test.mjs +389 -0
- package/lib/security/coerce-args.mjs +552 -0
- package/lib/security/coerce-args.test.mjs +281 -0
- package/lib/security/dangerous-tools.mjs +97 -0
- package/lib/security/dangerous-tools.test.mjs +68 -0
- package/lib/security/external-content.mjs +145 -0
- package/lib/security/external-content.test.mjs +67 -0
- package/lib/security/redact.mjs +592 -0
- package/lib/security/redact.test.mjs +441 -0
- package/lib/security/secret-equal.mjs +73 -0
- package/lib/security/secret-equal.test.mjs +55 -0
- package/lib/session-permissions.mjs +101 -0
- package/lib/session-permissions.test.mjs +100 -0
- package/lib/setup/claude-probe.mjs +74 -0
- package/lib/setup/completeness.mjs +175 -0
- package/lib/setup/completeness.test.mjs +110 -0
- package/lib/setup/context-pack.mjs +173 -0
- package/lib/setup/context-pack.test.mjs +89 -0
- package/lib/setup/enrich.mjs +277 -0
- package/lib/setup/enrich.test.mjs +115 -0
- package/lib/setup/enroll-from-cohort.mjs +441 -0
- package/lib/setup/enroll-from-cohort.test.mjs +233 -0
- package/lib/setup/integration.test.mjs +162 -0
- package/lib/setup/io.mjs +360 -0
- package/lib/setup/io.test.mjs +77 -0
- package/lib/setup/run-generator.mjs +81 -0
- package/lib/setup/runner.mjs +244 -0
- package/lib/setup/runner.test.mjs +132 -0
- package/lib/setup/sections/comms.mjs +173 -0
- package/lib/setup/sections/company.mjs +120 -0
- package/lib/setup/sections/enrich.mjs +138 -0
- package/lib/setup/sections/identity.mjs +182 -0
- package/lib/setup/sections/identity.test.mjs +140 -0
- package/lib/setup/sections/learning.mjs +153 -0
- package/lib/setup/sections/learning.test.mjs +81 -0
- package/lib/setup/sections/messaging.mjs +219 -0
- package/lib/setup/sections/messaging.test.mjs +127 -0
- package/lib/setup/sections/model.mjs +102 -0
- package/lib/setup/sections/operating-model.mjs +78 -0
- package/lib/setup/sections/org.mjs +475 -0
- package/lib/setup/sections/org.test.mjs +313 -0
- package/lib/setup/sections/orgmail.mjs +173 -0
- package/lib/setup/sections/orgmail.test.mjs +118 -0
- package/lib/setup/sections/recovery.mjs +159 -0
- package/lib/setup/sections/recovery.test.mjs +98 -0
- package/lib/setup/sections/tools.mjs +132 -0
- package/lib/setup/sections/verify.mjs +97 -0
- package/lib/setup/sot.mjs +205 -0
- package/lib/setup/sot.test.mjs +81 -0
- package/lib/setup/state.mjs +151 -0
- package/lib/setup/state.test.mjs +92 -0
- package/lib/singleton.js +229 -0
- package/lib/singleton.test.mjs +135 -0
- package/lib/telemetry/alerts.mjs +216 -0
- package/lib/telemetry/alerts.test.mjs +109 -0
- package/lib/telemetry/collect.mjs +512 -0
- package/lib/telemetry/collect.test.mjs +202 -0
- package/lib/tool-definitions-integration.test.mjs +83 -0
- package/lib/tool-definitions.js +738 -0
- package/lib/tool-definitions.test.mjs +437 -0
- package/lib/util/fetch-timeout.mjs +136 -0
- package/lib/util/fetch-timeout.test.mjs +202 -0
- package/lib/util/reconnect.mjs +343 -0
- package/lib/util/reconnect.test.mjs +369 -0
- package/lib/util/unhandled.mjs +205 -0
- package/lib/util/unhandled.test.mjs +216 -0
- package/lib/voice/context-loader.mjs +466 -0
- package/lib/voice/index.mjs +100 -0
- package/lib/voice/openai-realtime.mjs +510 -0
- package/lib/voice/outbound.mjs +542 -0
- package/lib/voice/outbound.test.mjs +69 -0
- package/lib/voice/post-call-brief.mjs +428 -0
- package/lib/voice/provider.mjs +52 -0
- package/lib/voice/session-rotation.mjs +257 -0
- package/lib/voice/stt.mjs +161 -0
- package/lib/voice/stt.test.mjs +226 -0
- package/lib/voice/tool-bridge.mjs +370 -0
- package/lib/voice/tts.mjs +104 -0
- package/lib/voice/twilio-sip-bridge.mjs +288 -0
- package/lib/voice/voice.test.mjs +990 -0
- package/mcp/README.md +80 -0
- package/package.json +151 -0
- package/plugins/maestro-skills/plugin.json +139 -0
- package/plugins/maestro-skills/skills/agents-observe.md +110 -0
- package/plugins/maestro-skills/skills/board-deck.md +68 -0
- package/plugins/maestro-skills/skills/books-close.md +77 -0
- package/plugins/maestro-skills/skills/brand-steward.md +121 -0
- package/plugins/maestro-skills/skills/calendar-plan.md +57 -0
- package/plugins/maestro-skills/skills/call-working-sessions.md +124 -0
- package/plugins/maestro-skills/skills/ccxray-diagnostics.md +91 -0
- package/plugins/maestro-skills/skills/claude-pace.md +61 -0
- package/plugins/maestro-skills/skills/code-review-graph.md +99 -0
- package/plugins/maestro-skills/skills/crm-pipeline.md +65 -0
- package/plugins/maestro-skills/skills/decision-brief.md +89 -0
- package/plugins/maestro-skills/skills/directory-hygiene.md +125 -0
- package/plugins/maestro-skills/skills/draft-comms.md +84 -0
- package/plugins/maestro-skills/skills/evening-wrap.md +53 -0
- package/plugins/maestro-skills/skills/files-find.md +65 -0
- package/plugins/maestro-skills/skills/generative-ui.md +228 -0
- package/plugins/maestro-skills/skills/hiring-triage.md +74 -0
- package/plugins/maestro-skills/skills/inbox-triage.md +61 -0
- package/plugins/maestro-skills/skills/mail-triage.md +86 -0
- package/plugins/maestro-skills/skills/morning-brief.md +54 -0
- package/plugins/maestro-skills/skills/native-artifacts.md +157 -0
- package/plugins/maestro-skills/skills/org-board.md +133 -0
- package/plugins/maestro-skills/skills/org-credential.md +68 -0
- package/plugins/maestro-skills/skills/org-recall.md +81 -0
- package/plugins/maestro-skills/skills/pipeline-review.md +76 -0
- package/plugins/maestro-skills/skills/regulatory-status.md +81 -0
- package/plugins/maestro-skills/skills/router-why.md +78 -0
- package/plugins/maestro-skills/skills/schedule-meeting.md +91 -0
- package/plugins/maestro-skills/skills/session-search.md +71 -0
- package/plugins/maestro-skills/skills/set-reminder.md +93 -0
- package/plugins/maestro-skills/skills/slack-followup.md +64 -0
- package/plugins/maestro-skills/skills/team-activity.md +86 -0
- package/plugins/maestro-skills/skills/weekly-memo.md +70 -0
- package/policies/action-classification.yaml +114 -0
- package/policies/ai-disclosure.yaml +294 -0
- package/policies/communication-style.md +139 -0
- package/policies/information-barriers.yaml +118 -0
- package/policies/prompt-injection-defence.yaml +138 -0
- package/public/assets/icon-dark.png +0 -0
- package/public/assets/icon-dark.svg +9 -0
- package/public/assets/icon-light.svg +9 -0
- package/public/assets/logo-dark.svg +15 -0
- package/public/assets/logo-light.svg +15 -0
- package/scaffold/.mcp.json +7 -0
- package/scaffold/CLAUDE.md +368 -0
- package/scaffold/config/agent.json +55 -0
- package/scaffold/config/agent.ts +76 -0
- package/scaffold/config/agent.ts.example +89 -0
- package/scaffold/config/alerts.yaml +23 -0
- package/scaffold/config/allowlist.yaml.example +25 -0
- package/scaffold/config/caller-id-map.yaml +46 -0
- package/scaffold/config/collective.yaml +49 -0
- package/scaffold/config/company.json +20 -0
- package/scaffold/config/known-agents.json +6 -0
- package/scaffold/config/learning.yaml +55 -0
- package/scaffold/config/model-routing.yaml.example +104 -0
- package/scaffold/config/org.yaml +25 -0
- package/scaffold/config/orgmail.yaml.example +19 -0
- package/scaffold/config/recovery.yaml +72 -0
- package/scaffold/config/secrets.yaml +27 -0
- package/scaffold/config/slack.yaml.example +35 -0
- package/scaffold/config/telegram.yaml.example +38 -0
- package/scaffold/config/voice.yaml.example +89 -0
- package/scaffold/config/whatsapp.yaml.example +39 -0
- package/schedules/README.md +49 -0
- package/schedules/triggers/backlog-executor.md +102 -0
- package/schedules/triggers/brand-steward.md +72 -0
- package/schedules/triggers/daily-evening-wrap.md +159 -0
- package/schedules/triggers/daily-midday-sweep.md +58 -0
- package/schedules/triggers/daily-morning-brief.md +55 -0
- package/schedules/triggers/directory-hygiene.md +81 -0
- package/schedules/triggers/dynamic-jobs.md +40 -0
- package/schedules/triggers/inbox-processor.md +115 -0
- package/schedules/triggers/meeting-action-capture.md +60 -0
- package/schedules/triggers/meeting-prep.md +69 -0
- package/schedules/triggers/messaging-inbound.md +50 -0
- package/schedules/triggers/org-pulse.md +24 -0
- package/schedules/triggers/quarterly-self-assessment.md +54 -0
- package/schedules/triggers/weekly-engineering-health.md +37 -0
- package/schedules/triggers/weekly-execution.md +65 -0
- package/schedules/triggers/weekly-hiring.md +53 -0
- package/schedules/triggers/weekly-priorities.md +38 -0
- package/schedules/triggers/weekly-strategic-memo.md +124 -0
- package/scripts/archive-email.sh +55 -0
- package/scripts/cadence/cadence-status.mjs +36 -0
- package/scripts/cadence/enqueue-cadence-tick.mjs +174 -0
- package/scripts/cadence/enqueue-cadence-tick.test.mjs +187 -0
- package/scripts/cadence/launchd-cadence-wrapper.sh +85 -0
- package/scripts/cadence/launchd-cloud-relay-wrapper.sh +95 -0
- package/scripts/cadence/launchd-socket-mode-wrapper.sh +95 -0
- package/scripts/ci/check-docs-accuracy.mjs +493 -0
- package/scripts/ci/check-docs-accuracy.test.mjs +409 -0
- package/scripts/ci/check-exports-exist.mjs +140 -0
- package/scripts/ci/check-files-exist.mjs +107 -0
- package/scripts/ci/check-no-build-artifacts.mjs +111 -0
- package/scripts/ci/check-no-build-artifacts.test.mjs +71 -0
- package/scripts/ci/check-no-confidential.mjs +198 -0
- package/scripts/ci/check-no-conflict-markers.mjs +169 -0
- package/scripts/ci/check-no-residual-identity.mjs +163 -0
- package/scripts/ci/check-no-residual-identity.test.mjs +89 -0
- package/scripts/ci/check-tarball-fidelity.mjs +205 -0
- package/scripts/ci/check-unresolved-tokens.mjs +83 -0
- package/scripts/ci/check.mjs +109 -0
- package/scripts/ci/check.test.mjs +194 -0
- package/scripts/ci/run-coverage.mjs +82 -0
- package/scripts/ci/run-tests.mjs +71 -0
- package/scripts/cloud-relay/README.md +59 -0
- package/scripts/cloud-relay/index.mjs +233 -0
- package/scripts/cloud-relay/package.json +15 -0
- package/scripts/cloud-relay/railway.json +13 -0
- package/scripts/cloud-relay/voice/README.md +94 -0
- package/scripts/cloud-relay/voice/package-lock.json +39 -0
- package/scripts/cloud-relay/voice/package.json +16 -0
- package/scripts/cloud-relay/voice/railway.json +13 -0
- package/scripts/cloud-relay/voice/server.mjs +532 -0
- package/scripts/collective/hook-runner.mjs +211 -0
- package/scripts/collective/hook-runner.test.mjs +90 -0
- package/scripts/collective/org-pulse.mjs +72 -0
- package/scripts/collective/org-sync.mjs +61 -0
- package/scripts/collective/recall.mjs +45 -0
- package/scripts/collective/who.mjs +30 -0
- package/scripts/comms-monitor.sh +288 -0
- package/scripts/configure-whatsapp-sandbox.sh +201 -0
- package/scripts/continuous-monitor.sh +91 -0
- package/scripts/cost/fleet-digest.mjs +407 -0
- package/scripts/cost/fleet-digest.test.mjs +207 -0
- package/scripts/cost/track-claude-usage.mjs +169 -0
- package/scripts/daemon/agent-daemon.mjs +989 -0
- package/scripts/daemon/agent-daemon.test.mjs +525 -0
- package/scripts/daemon/cadence-consumer-governance.test.mjs +220 -0
- package/scripts/daemon/cadence-consumer.mjs +1080 -0
- package/scripts/daemon/cadence-consumer.test.mjs +770 -0
- package/scripts/daemon/cadence-handlers.mjs +1121 -0
- package/scripts/daemon/cadence-handlers.test.mjs +617 -0
- package/scripts/daemon/classifier.mjs +704 -0
- package/scripts/daemon/classifier.test.mjs +238 -0
- package/scripts/daemon/classify-kind.mjs +54 -0
- package/scripts/daemon/classify-kind.test.mjs +40 -0
- package/scripts/daemon/context-compiler.mjs +605 -0
- package/scripts/daemon/context-compiler.test.mjs +300 -0
- package/scripts/daemon/dispatcher-cooldown.test.mjs +122 -0
- package/scripts/daemon/dispatcher-governance.test.mjs +886 -0
- package/scripts/daemon/dispatcher.mjs +1516 -0
- package/scripts/daemon/health.mjs +72 -0
- package/scripts/daemon/inbox-deferral.mjs +210 -0
- package/scripts/daemon/inbox-deferral.test.mjs +242 -0
- package/scripts/daemon/integration.test.mjs +149 -0
- package/scripts/daemon/launchd-wrapper-generic.sh +96 -0
- package/scripts/daemon/launchd-wrapper-slack-events.sh +37 -0
- package/scripts/daemon/launchd-wrapper.sh +91 -0
- package/scripts/daemon/lib/session-router.mjs +274 -0
- package/scripts/daemon/lib/session-router.test.mjs +295 -0
- package/scripts/daemon/maestro-daemon.mjs +275 -0
- package/scripts/daemon/prompt-builder.mjs +685 -0
- package/scripts/daemon/prompt-builder.test.mjs +213 -0
- package/scripts/daemon/responder.mjs +854 -0
- package/scripts/daemon/session-lock.mjs +721 -0
- package/scripts/daemon/session-lock.test.mjs +252 -0
- package/scripts/daemon/session-outcomes.mjs +640 -0
- package/scripts/daemon/session-outcomes.test.mjs +533 -0
- package/scripts/daemon/typing-registry.mjs +90 -0
- package/scripts/daemon/typing-registry.test.mjs +77 -0
- package/scripts/daemon/voice-webhook-server.mjs +804 -0
- package/scripts/decisions/capture-decision.mjs +116 -0
- package/scripts/disclosure_assessment.py +873 -0
- package/scripts/disclosure_boundaries.py +562 -0
- package/scripts/email-signature-principal.html +52 -0
- package/scripts/email-signature.html +60 -0
- package/scripts/email_quote_thread.py +167 -0
- package/scripts/email_thread_dedup.py +362 -0
- package/scripts/emergency-stop.sh +81 -0
- package/scripts/healthcheck.sh +116 -0
- package/scripts/hooks/block-mcp-cohort-send.sh +15 -0
- package/scripts/hooks/block-mcp-slack-send.sh +7 -0
- package/scripts/hooks/post-action-log.sh +126 -0
- package/scripts/hooks/pre-send-audit.sh +174 -0
- package/scripts/hooks/pre-send-audit.test.mjs +215 -0
- package/scripts/hooks/session-end-log.sh +27 -0
- package/scripts/hooks/session-start-banner.sh +115 -0
- package/scripts/huddle/audio-bridge.mjs +664 -0
- package/scripts/huddle/boot-slack-cdp.sh +102 -0
- package/scripts/huddle/huddle-controller.mjs +942 -0
- package/scripts/huddle/huddle-server.mjs +1229 -0
- package/scripts/huddle/launch-slack.sh +232 -0
- package/scripts/huddle/openai-realtime-bridge.mjs +462 -0
- package/scripts/huddle/package-lock.json +62 -0
- package/scripts/huddle/package.json +22 -0
- package/scripts/huddle/setup-audio.sh +239 -0
- package/scripts/huddle/start-call.mjs +318 -0
- package/scripts/huddle/test-pipeline.mjs +263 -0
- package/scripts/learning/consolidate-skills.mjs +72 -0
- package/scripts/learning/session-search.mjs +125 -0
- package/scripts/llm_email_dedup.py +442 -0
- package/scripts/local-triggers/generate-plists.sh +432 -0
- package/scripts/local-triggers/generate-plists.test.mjs +413 -0
- package/scripts/local-triggers/install-all.sh +49 -0
- package/scripts/local-triggers/plists/.gitkeep +0 -0
- package/scripts/local-triggers/run-trigger.sh +63 -0
- package/scripts/local-triggers/templates/rag-reindex.plist.template +47 -0
- package/scripts/local-triggers/templates/voice-relay-poller.plist.template +54 -0
- package/scripts/local-triggers/templates/voice-tunnel.plist.template +55 -0
- package/scripts/local-triggers/templates/voice-webhook.plist.template +51 -0
- package/scripts/maintenance/backup-to-cloud.sh +124 -0
- package/scripts/maintenance/health-check.sh +377 -0
- package/scripts/media-generation/README.md +105 -0
- package/scripts/media-generation/gemini-image-client.mjs +173 -0
- package/scripts/media-generation/generate-assets.mjs +289 -0
- package/scripts/media-generation/veo-video-client.mjs +219 -0
- package/scripts/org/send-orgmail.mjs +227 -0
- package/scripts/outbound-dedup-cleanup.sh +43 -0
- package/scripts/outbound-dedup.sh +477 -0
- package/scripts/outbound_dedup.py +115 -0
- package/scripts/parse-voice-transcript.mjs +481 -0
- package/scripts/pdf-generation/README.md +63 -0
- package/scripts/pdf-generation/build-document.mjs +247 -0
- package/scripts/pdf-generation/templates/board-pack.latex +136 -0
- package/scripts/pdf-generation/templates/corporate-letter.latex +126 -0
- package/scripts/pdf-generation/templates/memo.latex +114 -0
- package/scripts/poll-slack-events.sh +35 -0
- package/scripts/poller/calendar-poller.mjs +12 -0
- package/scripts/poller/gmail-poller.mjs +192 -0
- package/scripts/poller/imap-client.mjs +289 -0
- package/scripts/poller/inbox-scan-poller.mjs +156 -0
- package/scripts/poller/inbox-scan-poller.test.mjs +231 -0
- package/scripts/poller/index.mjs +73 -0
- package/scripts/poller/intra-session-check.mjs +285 -0
- package/scripts/poller/lib/cloud-relay-dedup.mjs +88 -0
- package/scripts/poller/lib/cloud-relay-dedup.test.mjs +133 -0
- package/scripts/poller/lib/slash-command-handlers.mjs +177 -0
- package/scripts/poller/secondary-gmail-poller.mjs +132 -0
- package/scripts/poller/slack-cloud-relay-client.mjs +368 -0
- package/scripts/poller/slack-poller.mjs +854 -0
- package/scripts/poller/slack-socket-mode.mjs +917 -0
- package/scripts/poller/slack-socket-mode.test.mjs +753 -0
- package/scripts/poller/trigger.mjs +75 -0
- package/scripts/poller/utils.mjs +371 -0
- package/scripts/poller/voice-cloud-relay-client.mjs +179 -0
- package/scripts/poller/voice-poller.mjs +236 -0
- package/scripts/poller-launchd/install.sh +66 -0
- package/scripts/poller-launchd/poller.plist.template +40 -0
- package/scripts/poller-launchd/whatsapp-handler.plist.template +39 -0
- package/scripts/post-interaction-indexer.py +1598 -0
- package/scripts/pre-draft-context.py +994 -0
- package/scripts/pre_draft_lookup.py +258 -0
- package/scripts/rag/build-index.mjs +47 -0
- package/scripts/rag/ingest.mjs +111 -0
- package/scripts/rag/search.mjs +119 -0
- package/scripts/rag-indexer.py +629 -0
- package/scripts/restore-from-backup.sh +248 -0
- package/scripts/restore-from-backup.test.mjs +178 -0
- package/scripts/resume-operations.sh +80 -0
- package/scripts/search-secondary-inbox.py +181 -0
- package/scripts/secondary-inbox-poller.py +437 -0
- package/scripts/self-optimization/compute-metrics.py +398 -0
- package/scripts/send-email-as-principal.py +369 -0
- package/scripts/send-email-threaded.py +392 -0
- package/scripts/send-email-with-attachment.py +377 -0
- package/scripts/send-email.sh +131 -0
- package/scripts/send-sms.sh +175 -0
- package/scripts/send-whatsapp.sh +292 -0
- package/scripts/session-start.sh +106 -0
- package/scripts/setup/boot-claude-session.sh +94 -0
- package/scripts/setup/configure-macos.sh +674 -0
- package/scripts/setup/configure-twilio-sip-trunk.mjs +207 -0
- package/scripts/setup/configure-voice-tunnel.mjs +182 -0
- package/scripts/setup/generate-agent-env.mjs +92 -0
- package/scripts/setup/generate-agent-package-json.mjs +222 -0
- package/scripts/setup/generate-agent-package-json.test.mjs +143 -0
- package/scripts/setup/generate-autonomy.mjs +60 -0
- package/scripts/setup/generate-backlog.mjs +101 -0
- package/scripts/setup/generate-cadences.mjs +92 -0
- package/scripts/setup/generate-capability.mjs +162 -0
- package/scripts/setup/generate-charter.mjs +76 -0
- package/scripts/setup/generate-comms.mjs +59 -0
- package/scripts/setup/generate-company.mjs +92 -0
- package/scripts/setup/init-agent.sh +547 -0
- package/scripts/setup/init-agent.test.mjs +151 -0
- package/scripts/setup/init-archetype.mjs +74 -0
- package/scripts/setup/init-backup.mjs +54 -0
- package/scripts/setup/init-cadence-bus.mjs +60 -0
- package/scripts/setup/init-channel-bus.mjs +46 -0
- package/scripts/setup/init-cost-tracking.mjs +45 -0
- package/scripts/setup/init-decision-capture.mjs +66 -0
- package/scripts/setup/init-known-agents.mjs +57 -0
- package/scripts/setup/init-learning.mjs +70 -0
- package/scripts/setup/init-memory-executive.mjs +45 -0
- package/scripts/setup/init-model-router.mjs +124 -0
- package/scripts/setup/init-rag.mjs +174 -0
- package/scripts/setup/init-session-router.mjs +38 -0
- package/scripts/setup/init-slack-socket-mode.mjs +260 -0
- package/scripts/setup/init-telegram.mjs +165 -0
- package/scripts/setup/init-voice-realtime.mjs +204 -0
- package/scripts/setup/init-whatsapp-baileys.mjs +77 -0
- package/scripts/setup/install-dev-tools.sh +150 -0
- package/scripts/setup/lib/install-plist.mjs +95 -0
- package/scripts/setup/migrate-agent-to-sot.mjs +192 -0
- package/scripts/setup/render-environment-yaml.mjs +133 -0
- package/scripts/slack-events-ctl.sh +177 -0
- package/scripts/slack-events-server.mjs +1045 -0
- package/scripts/slack-react.mjs +89 -0
- package/scripts/slack-responded.sh +232 -0
- package/scripts/slack-send.sh +287 -0
- package/scripts/slack-typing.mjs +196 -0
- package/scripts/slack-upload-v2.py +95 -0
- package/scripts/sms-handler.mjs +450 -0
- package/scripts/spawn-session.sh +120 -0
- package/scripts/sync-protocol.mjs +217 -0
- package/scripts/system-verify.sh +184 -0
- package/scripts/test-email-thread-dedup.py +239 -0
- package/scripts/test-information-barriers.py +484 -0
- package/scripts/test-llm-email-dedup.py +251 -0
- package/scripts/test-pre-draft-integration.py +203 -0
- package/scripts/test-rag-phase2.sh +442 -0
- package/scripts/test-rag-search.sh +251 -0
- package/scripts/test-voice-parser.mjs +316 -0
- package/scripts/user-context-search.py +659 -0
- package/scripts/validate_outbound.py +1504 -0
- package/scripts/watchdog/ai.maestro.memory-watchdog.plist +41 -0
- package/scripts/watchdog/force-reboot.sh +157 -0
- package/scripts/watchdog/memory-watchdog.sh +473 -0
- package/scripts/whatsapp-handler.mjs +538 -0
- package/teams/desktop-operations.yaml +34 -0
- package/teams/executive-office.yaml +27 -0
- package/teams/legal-and-regulatory.yaml +24 -0
- package/teams/platform-and-engineering.yaml +23 -0
- package/teams/strategy-and-growth.yaml +29 -0
- package/workflows/continuous/backlog-executor.yaml +141 -0
- package/workflows/continuous/inbound-monitor.yaml +168 -0
- package/workflows/daily/applicant-triage.yaml +197 -0
- package/workflows/daily/comms-triage.yaml +80 -0
- package/workflows/daily/evening-wrap.yaml +105 -0
- package/workflows/daily/morning-brief.yaml +164 -0
- package/workflows/daily/slack-followup-sweep.yaml +87 -0
- package/workflows/event-driven/README.md +50 -0
- package/workflows/event-driven/agent-failure-investigation.yaml +137 -0
- package/workflows/event-driven/pr-review.yaml +107 -0
- package/workflows/monthly/board-readiness.yaml +76 -0
- package/workflows/quarterly/strategic-scenario-analysis.yaml +85 -0
- package/workflows/session-protocol.md +171 -0
- package/workflows/weekly/engineering-health.yaml +154 -0
- package/workflows/weekly/hiring-review.yaml +169 -0
- package/workflows/weekly/rollup-pipeline-review.yaml +76 -0
- package/workflows/weekly/strategic-memo.yaml +79 -0
|
@@ -0,0 +1,205 @@
|
|
|
1
|
+
# Runbook — Backup and Restore
|
|
2
|
+
|
|
3
|
+
For an operator who must (a) confirm an agent's backups are real and current, or
|
|
4
|
+
(b) rebuild a dead Mac mini and recover its secrets and state. A Maestro agent's
|
|
5
|
+
durable state (`state/`: queues, cost ledger, learning DB, allowlists), its logs,
|
|
6
|
+
and its `.env` secrets live on a single disk. Losing the disk without a working
|
|
7
|
+
backup loses the agent's local history; this runbook is how you avoid that and
|
|
8
|
+
how you recover when it happens.
|
|
9
|
+
|
|
10
|
+
This complements `docs/runbooks/mac-mini-bootstrap.md` (fresh provisioning) and
|
|
11
|
+
`docs/runbooks/incident-response.md` (a compromised, not dead, machine). For a
|
|
12
|
+
compromised machine, do NOT restore onto the same hardware — rebuild on a clean
|
|
13
|
+
box.
|
|
14
|
+
|
|
15
|
+
## What is backed up, and the honest gaps
|
|
16
|
+
|
|
17
|
+
Off-machine backup is driven by `scripts/maintenance/backup-to-cloud.sh`, reading
|
|
18
|
+
`.maestro/backup-config.yaml`, run daily via launchd. Configure it with
|
|
19
|
+
`maestro init backup-replication --apply` (`scripts/setup/init-backup.mjs`).
|
|
20
|
+
|
|
21
|
+
Know these caveats before you rely on it:
|
|
22
|
+
|
|
23
|
+
- **Backup is opt-in and silent when off.** If `.maestro/backup-config.yaml` is
|
|
24
|
+
missing or `enabled: false`, the script exits 0 and does nothing. An agent with
|
|
25
|
+
no backup configured looks identical to a healthy one in casual inspection —
|
|
26
|
+
check explicitly (below).
|
|
27
|
+
- **Coverage historically excluded `logs/` and the audit records.** Confirm your
|
|
28
|
+
config's include paths cover `state/`, `logs/`, and `.maestro/`. If `logs/` is
|
|
29
|
+
excluded, a dead machine loses that agent's local compliance history — the
|
|
30
|
+
org-server export (next section) is your compensating control.
|
|
31
|
+
- **`retention_days` may not be enforced** by older builds; do not assume old
|
|
32
|
+
archives are pruned.
|
|
33
|
+
- **Archives may be unencrypted** unless you enabled client-side encryption.
|
|
34
|
+
Treat the backup bucket as sensitive, restrict access, and prefer an encrypted
|
|
35
|
+
bucket and encrypted archives.
|
|
36
|
+
|
|
37
|
+
The org server is your fleet-as-DR layer for the audit record: every
|
|
38
|
+
side-effecting action is a hash-chained `org_events` row at Cohort, exportable
|
|
39
|
+
as OCSF/OTel (`cohort export-audit`). So even if a single mini's local `logs/`
|
|
40
|
+
are lost, the authoritative cross-agent audit trail survives on the server. The
|
|
41
|
+
local logs hold richer detail (transcripts, raw messages); the server holds the
|
|
42
|
+
non-repudiable record.
|
|
43
|
+
|
|
44
|
+
## Confirm a backup is real and current (do this before you need it)
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
cd ~/<agent>-ai
|
|
48
|
+
|
|
49
|
+
# 1. Is backup configured and enabled?
|
|
50
|
+
test -f .maestro/backup-config.yaml && grep -E '^enabled:' .maestro/backup-config.yaml
|
|
51
|
+
|
|
52
|
+
# 2. Did the last run succeed, and when?
|
|
53
|
+
tail -n 20 logs/maintenance/backup.log
|
|
54
|
+
|
|
55
|
+
# 3. Does the configured destination actually have today's archive?
|
|
56
|
+
# (provider-specific; e.g. for GCS:)
|
|
57
|
+
# gcloud storage ls "gs://<bucket>/<prefix>/$(date -u +%F)/"
|
|
58
|
+
|
|
59
|
+
# 4. doctor surfaces backup freshness in the governance posture.
|
|
60
|
+
maestro doctor
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
If `maestro doctor` does not flag a missing/stale backup red, do not assume the
|
|
64
|
+
backup is fine — verify the destination has a recent archive. A backup you have
|
|
65
|
+
never restored from is a hypothesis, not a backup; periodically restore one agent
|
|
66
|
+
into a scratch directory to prove the archive is usable.
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## Restore a dead Mac mini
|
|
71
|
+
|
|
72
|
+
You are rebuilding an agent on replacement hardware. The order matters: provision
|
|
73
|
+
the host, recover secrets, restore state, re-establish org identity, then resume.
|
|
74
|
+
|
|
75
|
+
### 1. Provision the replacement host
|
|
76
|
+
|
|
77
|
+
Follow `docs/runbooks/mac-mini-bootstrap.md` to get a clean Mac mini to the point
|
|
78
|
+
where `~/maestro` is available and the agent repo is cloned to `~/<agent>-ai`.
|
|
79
|
+
Do not start the daemon yet.
|
|
80
|
+
|
|
81
|
+
### 2. Recover secrets
|
|
82
|
+
|
|
83
|
+
The `.env` is plaintext and there is no broker yet, so secret recovery is the
|
|
84
|
+
hardest step. In priority order:
|
|
85
|
+
|
|
86
|
+
- **From the backup archive**, if `.maestro/` and the agent's secret material
|
|
87
|
+
were included and the archive is encrypted at rest. Restore `.env` from the
|
|
88
|
+
archive, then immediately `chmod 600 .env`.
|
|
89
|
+
- **From the operator's escrow**, if your deployment keeps a sealed copy (e.g. in
|
|
90
|
+
a password manager entry per agent). This is the recommended belt-and-braces
|
|
91
|
+
until the secrets broker ships.
|
|
92
|
+
- **By re-issuing**, for anything you cannot recover: rotate the credential at
|
|
93
|
+
its source (Slack, WhatsApp, Gmail OAuth, relay) and write the new value. The
|
|
94
|
+
agent's Cohort device token specifically should be RE-MINTED, not restored —
|
|
95
|
+
see step 4.
|
|
96
|
+
|
|
97
|
+
If you are recovering after a *theft* (not just a hardware failure), treat every
|
|
98
|
+
secret as exposed and rotate per `docs/runbooks/incident-response.md` Scenario B
|
|
99
|
+
rather than restoring the old values.
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
cd ~/<agent>-ai
|
|
103
|
+
# after placing .env:
|
|
104
|
+
chmod 600 .env
|
|
105
|
+
stat -f '%Sp' .env # confirm -rw------- (0600)
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
### 3. Restore state from the latest archive
|
|
109
|
+
|
|
110
|
+
```bash
|
|
111
|
+
cd ~/<agent>-ai
|
|
112
|
+
# Pull the most recent archive from the configured destination into a temp dir,
|
|
113
|
+
# then unpack into the repo root so state/, logs/, and .maestro/ land in place.
|
|
114
|
+
# (provider-specific download; example for GCS:)
|
|
115
|
+
# gcloud storage cp "gs://<bucket>/<prefix>/<DATE>/<archive>" /tmp/restore.tar.gz
|
|
116
|
+
mkdir -p /tmp/restore && tar -xzf /tmp/restore.tar.gz -C /tmp/restore
|
|
117
|
+
# Inspect before overwriting:
|
|
118
|
+
ls /tmp/restore
|
|
119
|
+
# Then place state/ (and logs/ if backed up) into the repo:
|
|
120
|
+
cp -R /tmp/restore/state ./
|
|
121
|
+
cp -R /tmp/restore/logs ./ 2>/dev/null || true
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Sanity-check the restored state: the cost ledger and learning DB are SQLite
|
|
125
|
+
(`better-sqlite3`); a torn copy will fail to open. If `state/` is corrupt or
|
|
126
|
+
incomplete, the agent can still start — its durable queues will be empty, and it
|
|
127
|
+
re-establishes cursors — but you will have lost in-flight work. Note the gap in
|
|
128
|
+
the rebuild ticket.
|
|
129
|
+
|
|
130
|
+
### 4. Re-establish org-server identity
|
|
131
|
+
|
|
132
|
+
Do NOT restore the old device token. Mint a fresh one so the rebuilt agent has a
|
|
133
|
+
clean, attributable identity:
|
|
134
|
+
|
|
135
|
+
```bash
|
|
136
|
+
# On the operator station, against Cohort:
|
|
137
|
+
cohort --db "$COHORT_DB" pair approve <agentId> --scopes <scopes>
|
|
138
|
+
# This prints a new device token — write it into ~/<agent>-ai/.env on the mini.
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
If the dead agent was previously `admin.deactivate`d (e.g. it died during an
|
|
142
|
+
incident), `admin.reactivate` it first. Confirm the agent appears in the registry
|
|
143
|
+
with the expected owner, sponsor, autonomy tier, and scopes — a rebuild is a good
|
|
144
|
+
moment to re-certify access.
|
|
145
|
+
|
|
146
|
+
### 5. Resume operations
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
cd ~/<agent>-ai
|
|
150
|
+
maestro doctor # green governance posture before going live
|
|
151
|
+
./scripts/resume-operations.sh # health check, clear any stale .emergency-stop, load launchd
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Watch the first cadence ticks and the first inbound message end-to-end. Confirm
|
|
155
|
+
the agent heartbeats to the org server (`presence.beat`) and that its first
|
|
156
|
+
side-effecting action lands a row in the audit chain (`cohort doctor` chain
|
|
157
|
+
check passes).
|
|
158
|
+
|
|
159
|
+
### 6. Re-arm backup on the new machine
|
|
160
|
+
|
|
161
|
+
A rebuilt machine with no backup is the next dead machine.
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
cd ~/<agent>-ai
|
|
165
|
+
maestro init backup-replication --apply # if .maestro/backup-config.yaml absent
|
|
166
|
+
# confirm include paths cover state/, logs/, .maestro/; confirm encryption on.
|
|
167
|
+
maestro doctor # backup freshness should clear within a day
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
## Cohort fleet-as-DR exports
|
|
173
|
+
|
|
174
|
+
The org server is the disaster-recovery layer for the audit/compliance record and
|
|
175
|
+
for fleet metadata, independent of any single mini's backup.
|
|
176
|
+
|
|
177
|
+
- **Audit history** survives machine loss: `cohort export-audit --format ocsf`
|
|
178
|
+
(or `--format otel`) reconstructs every approval, decision, cost, and handoff
|
|
179
|
+
the lost agent participated in, with hash-chain provenance, from the server's
|
|
180
|
+
`org_events`. Run this as part of restore to confirm the lost agent's history
|
|
181
|
+
is intact server-side, and to fill any gap left by missing local `logs/`.
|
|
182
|
+
- **Identity and registry** survive: the agent's owner, sponsor, tier, scopes,
|
|
183
|
+
and credential bindings are server-side records re-bound at re-pair time, not
|
|
184
|
+
reconstructed from the dead disk.
|
|
185
|
+
- **Cost rollups** survive: per-agent/model/day spend reported via `cost.report`
|
|
186
|
+
is server-side, so financial accounting for the lost period is recoverable even
|
|
187
|
+
if the local ledger is gone.
|
|
188
|
+
|
|
189
|
+
Back up the Cohort database itself on its own schedule — it is the single point
|
|
190
|
+
whose loss takes the fleet's audit trail with it. Treat the Cohort DB and
|
|
191
|
+
`COHORT_SIGNING_SECRET` as tier-zero: encrypted backups, restricted access,
|
|
192
|
+
tested restore.
|
|
193
|
+
|
|
194
|
+
## Quick reference
|
|
195
|
+
|
|
196
|
+
| Need | Command |
|
|
197
|
+
|---|---|
|
|
198
|
+
| Is backup on? | `grep -E '^enabled:' .maestro/backup-config.yaml` |
|
|
199
|
+
| Last backup status | `tail logs/maintenance/backup.log` |
|
|
200
|
+
| Configure backup | `maestro init backup-replication --apply` |
|
|
201
|
+
| Governance posture incl. backup freshness | `maestro doctor` |
|
|
202
|
+
| Re-mint agent identity | `cohort pair approve <agentId> --scopes <...>` |
|
|
203
|
+
| Recover audit history | `cohort export-audit --format ocsf` |
|
|
204
|
+
| Resume after restore | `./scripts/resume-operations.sh` |
|
|
205
|
+
| Lock down `.env` | `chmod 600 .env` |
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
# Runbook — Cohort Cutover
|
|
2
|
+
|
|
3
|
+
For the operator moving the org server and the agent fleet from the Neolith
|
|
4
|
+
name to Cohort. The `cohort` branch of this repo carries the full text rename —
|
|
5
|
+
`services/neolith` → `services/cohort`, package `@neolith/agent-sdk` →
|
|
6
|
+
`@cohortapp/agent-sdk`, `COHORT_*` env names, `bin/cohort.mjs` — and this runbook
|
|
7
|
+
is the infrastructure the code change cannot reach: the Railway repoint, the
|
|
8
|
+
variable names, the npm scope, and the per-machine fleet update. The
|
|
9
|
+
product-app side of the same cutover (domains, OAuth, app releases) lives in
|
|
10
|
+
the `adapticai/hq` repo as `COHORT-RENAME.md`.
|
|
11
|
+
|
|
12
|
+
Companion runbooks: `docs/runbooks/fleet-operations.md` (rollout rings — reuse
|
|
13
|
+
them for every step here that touches minis), `docs/runbooks/backup-restore.md`,
|
|
14
|
+
`docs/runbooks/incident-response.md`. `DEPLOYMENT.md` describes the org-server
|
|
15
|
+
architecture this runbook repoints.
|
|
16
|
+
|
|
17
|
+
Order: npm scope first (additive), then the org server, then the avatar
|
|
18
|
+
worker, then the fleet, cleanup last.
|
|
19
|
+
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## 1. Org server on Railway
|
|
23
|
+
|
|
24
|
+
The service builds from this repo with the **repo root as build context** —
|
|
25
|
+
the Dockerfile reaches up for `lib/` (`DEPLOYMENT.md` §2.1). The rename moved
|
|
26
|
+
`services/neolith` → `services/cohort`, so make all of this **one** Settings
|
|
27
|
+
visit, not three:
|
|
28
|
+
|
|
29
|
+
1. **Variables.** `COHORT_*` is the canonical prefix, but every entry point —
|
|
30
|
+
`bin/maestro.mjs`, `bin/cohort-mcp.mjs`, `lib/org/client.mjs`, and the
|
|
31
|
+
server's own `bin/cohort.mjs` — first runs `applyBrandEnvCompat()`
|
|
32
|
+
(`lib/env-compat.mjs`), bridging `NEOLITH_*` ⇄ `COHORT_*`. The repoint
|
|
33
|
+
therefore works with the existing `NEOLITH_*` variables untouched. Add the
|
|
34
|
+
canonical spellings anyway and migrate onto them: `COHORT_ADMIN_TOKEN`,
|
|
35
|
+
`COHORT_SIGNING_KEY` / `COHORT_SIGNING_KEY_FILE`, `COHORT_CRED_KEY`,
|
|
36
|
+
`COHORT_DB` (SQLite pilot) or `DATABASE_URL` (Postgres), `COHORT_PORT`
|
|
37
|
+
(default 7470). Annotated list: `services/cohort/.env.example`. Delete the
|
|
38
|
+
legacy twins at cleanup (§5), not before.
|
|
39
|
+
2. **Source → Branch:** `main` → `cohort`.
|
|
40
|
+
3. **Config path, same action:** if the service pins a Railway Config File
|
|
41
|
+
(`services/neolith/railway.json`) or a Root Directory under
|
|
42
|
+
`services/neolith`, change it to `services/cohort/…` — otherwise the first
|
|
43
|
+
`cohort` build cannot find its config. The tracked
|
|
44
|
+
`services/cohort/railway.json` pins
|
|
45
|
+
`dockerfilePath: services/cohort/Dockerfile`,
|
|
46
|
+
`startCommand: node bin/cohort.mjs serve`, healthcheck `/v1/ops`.
|
|
47
|
+
|
|
48
|
+
Verify after the deploy: build green, `/v1/ops` healthy, agent heartbeats
|
|
49
|
+
landing, a `credential.lease` succeeds, and `org_events` keep chaining — the
|
|
50
|
+
audit hash chain is content-based, so a branch or name change never breaks it.
|
|
51
|
+
|
|
52
|
+
**Do not rename the Railway service in the same window.** Every fleet
|
|
53
|
+
machine's `config/org.yaml` pins `server.url` (e.g.
|
|
54
|
+
`https://neolith.up.railway.app`) plus the TLS fingerprint; renaming the
|
|
55
|
+
service changes the `*.up.railway.app` host and strands the fleet. Move the
|
|
56
|
+
fleet to a stable custom domain first (§2), then rename freely.
|
|
57
|
+
|
|
58
|
+
---
|
|
59
|
+
|
|
60
|
+
## 2. Custom domain for the org server
|
|
61
|
+
|
|
62
|
+
Railway → service → Settings → Networking: add `org.cohortapp.com`; create the
|
|
63
|
+
CNAME Railway displays at the `cohortapp.com` DNS host. Old and new hosts
|
|
64
|
+
serve in parallel — there is no flag day for the fleet.
|
|
65
|
+
|
|
66
|
+
Then roll `config/org.yaml` across machines through rings, never all at once:
|
|
67
|
+
|
|
68
|
+
- `server.url` → `https://org.cohortapp.com`
|
|
69
|
+
- `server.fingerprint` → re-pin against the new host's TLS cert
|
|
70
|
+
|
|
71
|
+
Pairing survives the move — agent identity is the device keypair, not the URL.
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
## 3. Avatar worker
|
|
76
|
+
|
|
77
|
+
`services/avatar` kept its name. The Railway avatars service needs only
|
|
78
|
+
**Source → Branch: `main` → `cohort`** — Root Directory stays
|
|
79
|
+
`services/avatar` and its `railway.json` (`dockerfilePath: Dockerfile`,
|
|
80
|
+
`startCommand: node bin/avatar-worker.mjs start`) is unchanged.
|
|
81
|
+
|
|
82
|
+
---
|
|
83
|
+
|
|
84
|
+
## 4. npm scope and the fleet
|
|
85
|
+
|
|
86
|
+
The package is now `@cohortapp/agent-sdk` (bins: `maestro`, `cohort`,
|
|
87
|
+
`cohort-mcp`, plus legacy aliases `neolith` and `neolith-mcp` so installed
|
|
88
|
+
launchd jobs survive the bump; `.npmrc` keeps the `@neolith:registry` mapping).
|
|
89
|
+
|
|
90
|
+
1. Create the npm org/scope **`cohort`** on npmjs.com.
|
|
91
|
+
2. First publish is manual — Trusted Publishing is configured on an existing
|
|
92
|
+
package's settings page. From a `cohort`-branch checkout:
|
|
93
|
+
`npm publish --access public` as an org member.
|
|
94
|
+
3. Wire Trusted Publishing: npmjs.com → `@cohortapp/agent-sdk` → Settings →
|
|
95
|
+
Trusted Publisher → GitHub Actions, org `adapticai`, repo `maestro`,
|
|
96
|
+
workflow `auto-publish-npm.yml`. The workflow is already OIDC-ready
|
|
97
|
+
(`id-token: write`, npm ≥ 11.5.1, strips the `.npmrc` authToken line,
|
|
98
|
+
no-ops on already-published versions).
|
|
99
|
+
4. Bump each mini through rings (`docs/runbooks/fleet-operations.md`), canary
|
|
100
|
+
first:
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
cd ~/<agent>-ai
|
|
104
|
+
maestro doctor # BEFORE posture
|
|
105
|
+
npm install @cohortapp/agent-sdk@latest
|
|
106
|
+
maestro doctor # AFTER must be no worse
|
|
107
|
+
./scripts/resume-operations.sh
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Machines still on `@neolith/agent-sdk` keep working against the repointed
|
|
111
|
+
server — the org protocol is frozen and name-agnostic, and `config/org.yaml`
|
|
112
|
+
pins the URL, not the package name — so the fleet bump can trail the server
|
|
113
|
+
cutover by days.
|
|
114
|
+
|
|
115
|
+
---
|
|
116
|
+
|
|
117
|
+
## 5. Cleanup
|
|
118
|
+
|
|
119
|
+
- Delete the `NEOLITH_*` variables from the Railway services once the
|
|
120
|
+
`COHORT_*` twins are set and a deploy has been verified reading them (the
|
|
121
|
+
env-compat bridge makes the overlap safe, not permanent).
|
|
122
|
+
- `npm deprecate @neolith/agent-sdk "renamed to @cohortapp/agent-sdk"` — if the
|
|
123
|
+
old scope was ever published.
|
|
124
|
+
- After the fleet is on `org.cohortapp.com`: optionally rename the Railway
|
|
125
|
+
service, then sweep anything pinned to the old `*.up.railway.app` host
|
|
126
|
+
(monitors, webhooks, stray `org.yaml` stragglers — the registry's presence
|
|
127
|
+
view shows who stopped heartbeating).
|
|
128
|
+
- Merge `cohort` → `main` and point the services back at `main`, mirroring the
|
|
129
|
+
hq-side sequence.
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
# Runbook — Fleet Operations
|
|
2
|
+
|
|
3
|
+
For an operator running 40–60 Maestro agents as a fleet. Covers rollout rings,
|
|
4
|
+
version skew, cross-agent triage, fleet-wide pause, and decommissioning an agent.
|
|
5
|
+
Maestro's per-machine substrate is strong; the fleet layer is the part you
|
|
6
|
+
operate by convention plus the Cohort org server. Where a fleet primitive is
|
|
7
|
+
still maturing, this runbook says so and gives the manual procedure.
|
|
8
|
+
|
|
9
|
+
Companion runbooks: `docs/runbooks/incident-response.md` (one bad agent),
|
|
10
|
+
`docs/runbooks/backup-restore.md` (one dead agent),
|
|
11
|
+
`docs/runbooks/recovery-and-failover.md` (operational recovery).
|
|
12
|
+
|
|
13
|
+
## The fleet is vendored per machine
|
|
14
|
+
|
|
15
|
+
Each agent runs `~/maestro` as a checkout/package on its own Mac mini. There is
|
|
16
|
+
no central push that updates all of them at once — upgrades are per machine and
|
|
17
|
+
drift is the default unless you manage it. This is why rings, a version floor, and
|
|
18
|
+
skew-watching matter operationally rather than as nice-to-haves.
|
|
19
|
+
|
|
20
|
+
The org server gives you the cross-machine views that the minis cannot give
|
|
21
|
+
themselves: registry/identity, presence/heartbeat, approvals, decisions, cost
|
|
22
|
+
rollups, policy distribution, and the audit export. Use it as the fleet's control
|
|
23
|
+
and observation plane; use SSH/runbooks on individual minis for anything the
|
|
24
|
+
server does not yet do remotely.
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## Rollout rings
|
|
29
|
+
|
|
30
|
+
Never upgrade all 40–60 agents at once. Promote through rings and let each ring
|
|
31
|
+
bake before the next.
|
|
32
|
+
|
|
33
|
+
1. **Canary (1–2 agents).** Pick low-stakes agents (not the CEO's, not a
|
|
34
|
+
compliance-adjacent one). Upgrade, then watch a full day of real traffic:
|
|
35
|
+
inbound handled, cadences firing, sends going out, heartbeats green, no new
|
|
36
|
+
doctor red flags, cost in band.
|
|
37
|
+
2. **Early ring (~10%).** A representative spread of archetypes. Bake 1–2 days.
|
|
38
|
+
Watch for archetype-specific breakage (a tool one role uses heavily that the
|
|
39
|
+
canary did not exercise).
|
|
40
|
+
3. **Broad ring (~50%).** Bake a day.
|
|
41
|
+
4. **Fleet (remainder).** Only after the broad ring is clean.
|
|
42
|
+
|
|
43
|
+
Per-agent upgrade on a mini:
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
cd ~/<agent>-ai
|
|
47
|
+
maestro doctor # capture the BEFORE posture
|
|
48
|
+
npm update @cohortapp/agent-sdk # or: git -C ~/maestro pull, for direct checkouts
|
|
49
|
+
maestro doctor # AFTER posture must be no worse
|
|
50
|
+
./scripts/resume-operations.sh # restart cleanly so the daemon picks up the new code
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Promotion gate between rings: zero new doctor red flags, heartbeats present for
|
|
54
|
+
every upgraded agent, no spike in the audit export's failure/refusal families, and
|
|
55
|
+
cost-per-session unchanged. If any ring regresses, STOP promoting and roll back
|
|
56
|
+
that ring before touching the next.
|
|
57
|
+
|
|
58
|
+
### Rollback
|
|
59
|
+
|
|
60
|
+
Pin the agent back to the last-known-good version and restart:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
cd ~/<agent>-ai
|
|
64
|
+
npm install @cohortapp/agent-sdk@<last-good-version> # or: git -C ~/maestro checkout <good-sha>
|
|
65
|
+
./scripts/resume-operations.sh
|
|
66
|
+
maestro doctor
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
For a direct `~/maestro` checkout, rollback is a `git checkout` of the known-good
|
|
70
|
+
SHA followed by a restart. Record the rollback and the regression in the rollout
|
|
71
|
+
ticket so the canary catches it next time.
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
## Version skew
|
|
76
|
+
|
|
77
|
+
A fleet mid-rollout is heterogeneous by design; the risk is *unbounded* skew and
|
|
78
|
+
agents below the floor.
|
|
79
|
+
|
|
80
|
+
- **Take inventory.** The registry should carry each agent's `maestroVersion` via
|
|
81
|
+
its heartbeat. Where that field is not yet populated, collect it manually:
|
|
82
|
+
`for a in <agents>; do ssh "$a" 'cd ~/<repo> && node -p "require(\"@cohortapp/agent-sdk/package.json\").version"'; done`.
|
|
83
|
+
- **Hold a version floor.** Define the minimum version any agent may run
|
|
84
|
+
(typically the last version that fixed a security or durability bug). Agents
|
|
85
|
+
below the floor are upgraded out of band, not left to the normal ring cadence —
|
|
86
|
+
a below-floor agent is an exposure, not just stale.
|
|
87
|
+
- **Bound the spread.** Do not let the newest and oldest agents differ by more
|
|
88
|
+
than one minor version in steady state. If a rollout stalls, either finish it
|
|
89
|
+
or roll the leading ring back to close the gap.
|
|
90
|
+
- **Watch for protocol skew with the server.** The agent↔Cohort protocol is
|
|
91
|
+
versioned. If you upgrade the server, confirm the oldest agent in the fleet
|
|
92
|
+
still speaks the protocol before you cut over; if not, raise the floor first.
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## Cross-agent triage
|
|
97
|
+
|
|
98
|
+
With 40–60 agents you cannot watch 40–60 Slack DM streams. Triage from the org
|
|
99
|
+
server's aggregate views, then drill into the one machine that needs it.
|
|
100
|
+
|
|
101
|
+
1. **Fleet health at a glance.** Presence/heartbeat tells you who is alive. A
|
|
102
|
+
stale heartbeat is the first signal — an agent that stopped beating is either
|
|
103
|
+
wedged, off, or its machine is down. (Where a `maestro fleet status` rollup is
|
|
104
|
+
available, use it; otherwise read presence from the server and `maestro
|
|
105
|
+
doctor` on suspects.)
|
|
106
|
+
2. **Behavioral signals, not single failures.** Agents fail silently by
|
|
107
|
+
completing with wrong output, so triage on patterns from the audit/cost
|
|
108
|
+
export: refusal spikes, per-tool failure rates above baseline, cost-per-session
|
|
109
|
+
p99 climbing, an agent that has gone quiet (no events) during its active hours.
|
|
110
|
+
Pull these from `cohort export-audit --format otel` (spans carry the
|
|
111
|
+
`gen_ai.*` and cost attributes) and the cost rollup.
|
|
112
|
+
3. **Correlate one interaction.** To reconstruct a single user interaction across
|
|
113
|
+
item → classification → session → send, follow the trace/decision identifiers:
|
|
114
|
+
the model-router `decision_id` joins a routing decision to its ledger row to
|
|
115
|
+
its resume marker, and the audit chain's `seq`/`row_hash` order the events.
|
|
116
|
+
This is how you answer "what did agent X actually do at 14:05" authoritatively.
|
|
117
|
+
4. **Drill to the machine.** Once you have the suspect, SSH in and run `maestro
|
|
118
|
+
doctor` for the one-screen governance posture (heartbeats, throttle, breaker,
|
|
119
|
+
budget band, stuck markers, permissions hygiene), then read its `logs/`.
|
|
120
|
+
|
|
121
|
+
---
|
|
122
|
+
|
|
123
|
+
## Fleet-wide pause (break-glass)
|
|
124
|
+
|
|
125
|
+
When you need everything to stop — a fleet-level security event, a bad upgrade
|
|
126
|
+
caught mid-rollout, a provider outage causing runaway retries.
|
|
127
|
+
|
|
128
|
+
There is no single command that pauses all minis today; pause is the sum of
|
|
129
|
+
per-agent kill switches plus server-side deactivation. Sequence:
|
|
130
|
+
|
|
131
|
+
1. **Cut coordination at the server.** Deactivate each agent (or, if your build
|
|
132
|
+
supports it, suspend the fleet); a deactivated agent's tokens are revoked and
|
|
133
|
+
it cannot coordinate or be impersonated:
|
|
134
|
+
```bash
|
|
135
|
+
for a in $(cat fleet-roster.txt); do
|
|
136
|
+
cohort --db "$COHORT_DB" admin deactivate "$a" --reason "fleet pause <ticket>"
|
|
137
|
+
done
|
|
138
|
+
```
|
|
139
|
+
2. **Halt execution on each mini** (the local break-glass), in parallel where you
|
|
140
|
+
have fan-out SSH:
|
|
141
|
+
```bash
|
|
142
|
+
# per machine:
|
|
143
|
+
cd ~/<agent>-ai && ./scripts/emergency-stop.sh
|
|
144
|
+
```
|
|
145
|
+
This drops `.emergency-stop`, unloads launchd jobs, and kills running sessions.
|
|
146
|
+
3. **Confirm silence.** Heartbeats stop; the audit export shows no new
|
|
147
|
+
side-effecting events. If any agent is still acting, it did not honor the stop
|
|
148
|
+
— investigate that machine directly.
|
|
149
|
+
|
|
150
|
+
Resume is deliberate and per agent (see "Resume" below). Do not script a
|
|
151
|
+
fleet-wide auto-resume; a pause exists because something needed a human decision.
|
|
152
|
+
|
|
153
|
+
Resume after the cause is cleared:
|
|
154
|
+
|
|
155
|
+
```bash
|
|
156
|
+
# per machine:
|
|
157
|
+
cd ~/<agent>-ai && ./scripts/resume-operations.sh
|
|
158
|
+
# then on the operator station, per agent:
|
|
159
|
+
cohort --db "$COHORT_DB" admin reactivate "$a"
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
---
|
|
163
|
+
|
|
164
|
+
## Decommissioning an agent
|
|
165
|
+
|
|
166
|
+
Retiring an agent permanently. The goal is no orphaned identity, no live
|
|
167
|
+
credentials, and a preserved audit trail.
|
|
168
|
+
|
|
169
|
+
1. **Stop it.** Local break-glass plus server deactivation:
|
|
170
|
+
```bash
|
|
171
|
+
cd ~/<agent>-ai && ./scripts/emergency-stop.sh
|
|
172
|
+
cohort --db "$COHORT_DB" admin deactivate <agentId> --reason "decommission"
|
|
173
|
+
```
|
|
174
|
+
Deactivation revokes its tokens — the identity-side half of the kill switch.
|
|
175
|
+
2. **Preserve its history before you touch the machine.** Export its audit trail
|
|
176
|
+
from the server (`cohort export-audit --format ocsf`, filtered to the agent)
|
|
177
|
+
and take a final off-machine backup of its `state/` and `logs/`
|
|
178
|
+
(`docs/runbooks/backup-restore.md`). A decommissioned agent's compliance
|
|
179
|
+
record must outlive the agent — retain per `docs/compliance/evidence-map.md`.
|
|
180
|
+
3. **Revoke channel access at the source.** Slack token, WhatsApp session, Gmail
|
|
181
|
+
OAuth, Telegram bot, relay secrets — revoke each in its provider so the retired
|
|
182
|
+
identity cannot send even if a credential lingers.
|
|
183
|
+
4. **Drain in-flight obligations.** Check the audit export's `handoff` and
|
|
184
|
+
`approval` families for anything the agent owed another agent or a human;
|
|
185
|
+
reassign or close those before the agent is gone, or they become orphaned.
|
|
186
|
+
5. **Wipe the machine.** Securely erase `.env` and `state/` on the mini (the
|
|
187
|
+
plaintext secrets especially). If the hardware is being redeployed for a new
|
|
188
|
+
agent, treat it as a fresh provision (`docs/runbooks/mac-mini-bootstrap.md`),
|
|
189
|
+
not a rename of the old one.
|
|
190
|
+
6. **Close the identity.** Mark the registry record decommissioned (its lifecycle
|
|
191
|
+
state), with the date and the operator who did it. Do not delete the record —
|
|
192
|
+
the audit trail references it; a deleted identity makes its history
|
|
193
|
+
un-attributable.
|
|
194
|
+
7. **Notify.** If the agent was HR-adjacent or interacted with people who relied
|
|
195
|
+
on it, tell those people it is retired (and who to talk to instead) — the
|
|
196
|
+
deployer's transparency duty does not end when the agent does.
|
|
197
|
+
|
|
198
|
+
A decommissioning is done when: the agent cannot authenticate, cannot send on any
|
|
199
|
+
channel, has no live launchd jobs, its history is exported and retained, its
|
|
200
|
+
registry record is marked retired (not deleted), and affected humans are notified.
|