agy-squad 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/full-review/SKILL.md +89 -0
- package/.agents/skills/full-review/references/evidence-intake.md +146 -0
- package/.agents/skills/full-review/references/report-template.md +216 -0
- package/.agents/skills/full-review/references/review-roles.md +91 -0
- package/.agents/skills/pr-review/SKILL.md +60 -0
- package/.agents/skills/pr-review/references/pr-review-standard.md +163 -0
- package/.agents/skills/self-diagnosis/SKILL.md +63 -0
- package/.agents/skills/self-diagnosis/references/execution-and-teardown.md +71 -0
- package/.agents/skills/self-diagnosis/references/optimization-and-review-rubric.md +61 -0
- package/.agents/skills/self-diagnosis/resources/comparative-research-manifest.json +42 -0
- package/.agents/skills/self-diagnosis/resources/self-diagnosis-manifest.json +144 -0
- package/.agents/skills/self-diagnosis/scripts/diagnose.mjs +1971 -0
- package/.agents/skills/swarm-governance/SKILL.md +59 -0
- package/.agents/skills/swarm-governance/references/bundling-and-wire-format.md +88 -0
- package/.agents/skills/swarm-governance/references/intake-and-planning.md +120 -0
- package/.agents/skills/swarm-governance/references/large-swarms.md +39 -0
- package/CHANGELOG.md +16 -0
- package/LICENSE +21 -0
- package/README.md +156 -2
- package/bin/agy-squad.mjs +293 -0
- package/docs/assets/architecture-overview.svg +39 -0
- package/docs/assets/manifest-and-modes.svg +39 -0
- package/docs/assets/role-and-verdict-gate.svg +39 -0
- package/docs/assets/skill-role-workflows.svg +39 -0
- package/docs/assets/swarm-lifecycle.svg +39 -0
- package/docs/benchmarks.md +102 -0
- package/docs/concepts.md +81 -0
- package/docs/core-workflows.md +128 -0
- package/docs/getting-started.md +116 -0
- package/docs/index.md +47 -0
- package/docs/plans/LEDGER.md +24 -0
- package/docs/plans/comparative-harness-architecture-retest.md +1174 -0
- package/docs/plans/comparative-harness-architecture-teamwork.md +1170 -0
- package/docs/plans/comparative-harness-architecture.md +256 -0
- package/docs/plans/native-enforcement-ledger.md +126 -0
- package/docs/plans/pr3-option-b-fixes.md +803 -0
- package/docs/plans/wsl-cli-verification.md +55 -0
- package/docs/reviews/post-pr5-governance-review.md +221 -0
- package/docs/reviews/pr1-code-review-ledger.md +581 -0
- package/docs/reviews/pr1-production-readiness-review.md +764 -0
- package/docs/reviews/pr2-documentation-review-ledger.md +380 -0
- package/docs/reviews/pr3-squad-alignment-full-review.md +1021 -0
- package/docs/reviews/pr4-swarm-governance-pivot-review.md +593 -0
- package/docs/reviews/pr5-full-review-direction.md +330 -0
- package/docs/reviews/pr5-independent-review.md +266 -0
- package/docs/reviews/pr5-layered-swarm-review.md +513 -0
- package/docs/reviews/pr5-phase2-swarm-planner-review.md +470 -0
- package/docs/reviews/pr5-reconciled-review.md +573 -0
- package/docs/reviews/rebuild-wsl-dispatch-review-r2.md +427 -0
- package/docs/reviews/rebuild-wsl-dispatch-review.md +552 -0
- package/docs/roles-and-workflows.md +115 -0
- package/docs/stopping-agents.md +39 -0
- package/docs/tools.md +122 -0
- package/install.ps1 +5 -0
- package/install.sh +4 -0
- package/mcp/briefs.mjs +202 -0
- package/mcp/dispatch.mjs +280 -0
- package/mcp/handoffs.mjs +814 -0
- package/mcp/index.js +322 -0
- package/mcp/planner.mjs +755 -0
- package/mcp/verdicts.mjs +224 -0
- package/package.json +57 -4
- package/plugin.json +20 -0
- package/scripts/install-register.mjs +510 -0
- package/scripts/release-preflight.mjs +142 -0
|
@@ -0,0 +1,1174 @@
|
|
|
1
|
+
# Comparative Multi-Agent Harness Architecture: Systems Engineering Analysis, Quantitative Rubrics, and Protocol Tracing
|
|
2
|
+
|
|
3
|
+
> [!NOTE]
|
|
4
|
+
> **Historical (pre-Phase 2).** Scores, teardown and `PGID` kill-control criteria, and launcher references in this document describe Agy-Squad at decisions `L1`–`L41` (Phase 1: PTY launcher, managed-session teardown). PR #5 retired the launcher and managed-session teardown (`L17`; `L42`/`L43`). Do not cite these scores as describing the current architecture.
|
|
5
|
+
|
|
6
|
+
**Document ID:** `PLAN-COMPARATIVE-HARNESS-ARCHITECTURE-RETEST-2026-11-10`
|
|
7
|
+
**Publication Standard:** Agy-Squad PR Review Standard `v1.3.0` & Architectural Decision Ledger (`L1`–`L41`)
|
|
8
|
+
**Target File:** `docs/plans/comparative-harness-architecture-retest.md`
|
|
9
|
+
**Author:** Harness Comparative Architecture Lead
|
|
10
|
+
**Squad ID:** `harness_comparative_audit`
|
|
11
|
+
**Specialist Audits Synthesized:**
|
|
12
|
+
1. Specialist 1 (`scratch/agy-squad/squad-harness_comparative_audit/specialist-1/report.md`): General Workflow & Graph Harnesses (MetaGPT, CrewAI, AutoGen / Swarm, LangGraph)
|
|
13
|
+
2. Specialist 2 (`scratch/agy-squad/squad-harness_comparative_audit/specialist-2/report.md`): Autonomous Coding & SWE Harnesses (Devin / Sol, Claude Code / SWE-agent, Antigravity Native /teamwork, Agy-Squad)
|
|
14
|
+
**Verification Baseline:** Empirical Event-Relay Benchmark (`benchmarks/teamwork-vs-squad/`), Automated Test Suites (`tests/`), and Live Subagent Transcript Forensics
|
|
15
|
+
**Date:** November 10, 2026
|
|
16
|
+
**Status:** PUBLICATION-GRADE / FINAL / PASS
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## Executive Abstract
|
|
21
|
+
|
|
22
|
+
Multi-agent coordination and autonomous software engineering (SWE) harnesses have evolved rapidly from single-turn code generation prompts into long-horizon, autonomous multi-turn loops. However, the systems-engineering foundation of modern multi-agent coordination remains fragmented. Most contemporary architectures conflate the **Inter-Process Communication (IPC) signaling wire** with the **evidentiary storage channel**. By transmitting complete document deliverables, conversational transcripts, execution scratchpads, and raw compiler dumps directly through model prompt contexts, modern harnesses induce quadratic ($O(M^2)$) or monotonic ($O(N \cdot |S|)$) context inflation, attention dilution ("lost-in-the-middle"), and prompt-cache invalidation. Furthermore, artifact verification in these frameworks relies on non-deterministic, high-token LLM-on-LLM critique rather than tamper-evident cryptographic contracts.
|
|
23
|
+
|
|
24
|
+
This dossier delivers a comprehensive, evidence-backed comparative architectural analysis of **Agy-Squad** against seven state-of-the-art multi-agent harnesses across two audited cohorts:
|
|
25
|
+
1. **General Workflow & Graph Harnesses:** MetaGPT (SOP-driven waterfall), CrewAI (role-playing collectives), AutoGen / OpenAI Swarm (conversational broadcast and stateless swarms), and LangGraph (cyclic statechart DAGs).
|
|
26
|
+
2. **Autonomous Coding & SWE Harnesses:** Devin / Cognition Sol (cloud microVM workspace), Claude Code & SWE-agent (Agent-Computer Interface [ACI] and interactive terminal loops), Google Antigravity Native `/teamwork` (hierarchical swarm fleets), and Agy-Squad (Pointer-Over-Wire + RTK).
|
|
27
|
+
|
|
28
|
+
We establish a **5-Axis Quantitative Evaluation Rubric** (Token & Context Preservation, Evidence Traceability & Cryptographic Proof, Failure Isolation & Blast Radius, Orchestration Topology Flexibility, and OS & Execution Boundary Safety) scoring all eight harnesses. We formalize mathematical token/byte complexity models, provide verbatim protocol traces comparing full-transcript chatter against Agy-Squad's $\le 500$-byte Pointer-Over-Wire receipts, and synthesize an exhaustive architectural trade-off matrix articulating the Pareto frontier, strengths, and strategic yield points.
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## Table of Contents
|
|
33
|
+
|
|
34
|
+
1. [Executive Framing & The Systems Engineering Trilemma](#1-executive-framing--the-systems-engineering-trilemma)
|
|
35
|
+
- 1.1 [The Multi-Agent Orchestration Trilemma](#11-the-multi-agent-orchestration-trilemma)
|
|
36
|
+
- 1.2 [Signaling vs. Storage Conflation: The Root Architectural Pathology](#12-signaling-vs-storage-conflation-the-root-architectural-pathology)
|
|
37
|
+
- 1.3 [The Agy-Squad Architectural Synthesis](#13-the-agy-squad-architectural-synthesis)
|
|
38
|
+
2. [Audited Target Group 1: General Workflow & Graph Multi-Agent Harnesses](#2-audited-target-group-1-general-workflow--graph-multi-agent-harnesses)
|
|
39
|
+
- 2.1 [MetaGPT: Standard Operating Procedure (SOP) Driven Waterfall](#21-metagpt-standard-operating-procedure-sop-driven-waterfall)
|
|
40
|
+
- 2.2 [CrewAI: Role-Playing Collectives & Chained Delegation](#22-crewai-role-playing-collectives--chained-delegation)
|
|
41
|
+
- 2.3 [AutoGen & OpenAI Swarm: Conversational Broadcast & Stateless Swarms](#23-autogen--openai-swarm-conversational-broadcast--stateless-swarms)
|
|
42
|
+
- 2.4 [LangGraph: Statecharts, Controlled Cyclic Graphs, & Channel Reducers](#24-langgraph-statecharts-controlled-cyclic-graphs--channel-reducers)
|
|
43
|
+
3. [Audited Target Group 2: Autonomous Coding & SWE Harnesses](#3-audited-target-group-2-autonomous-coding--swe-harnesses)
|
|
44
|
+
- 3.1 [Devin / Cognition Sol: Cloud MicroVMs & Snapshot-Resume Workspace](#31-devin--cognition-sol-cloud-microvms--snapshot-resume-workspace)
|
|
45
|
+
- 3.2 [Claude Code & SWE-agent: The Agent-Computer Interface (ACI)](#32-claude-code--swe-agent-the-agent-computer-interface-aci)
|
|
46
|
+
- 3.3 [Google Antigravity Native `/teamwork`: Hierarchical Swarms & Reactive Wakeups](#33-google-antigravity-native-teamwork-hierarchical-swarms--reactive-wakeups)
|
|
47
|
+
- 3.4 [Agy-Squad Reference Baseline: Staged Manifests & Cryptographic Governance](#34-agy-squad-reference-baseline-staged-manifests--cryptographic-governance)
|
|
48
|
+
4. [5-Axis Quantitative Evaluation Rubric & Comparative Scores](#4-5-axis-quantitative-evaluation-rubric--comparative-scores)
|
|
49
|
+
- 4.1 [Rubric Metric Definitions & 1.0–5.0 Scale Anchors](#41-rubric-metric-definitions--1050-scale-anchors)
|
|
50
|
+
- 4.2 [Comprehensive 8-Harness Quantitative Scoring Matrix](#42-comprehensive-8-harness-quantitative-scoring-matrix)
|
|
51
|
+
- 4.3 [Detailed Architectural Justifications across All 8 Harnesses](#43-detailed-architectural-justifications-across-all-8-harnesses)
|
|
52
|
+
5. [Empirical Evidence & Protocol Tracing (R3)](#5-empirical-evidence--protocol-tracing-r3)
|
|
53
|
+
- 5.1 [Formal Mathematical Complexity Models ($O(N^2)$ vs $O(N \cdot |S|)$ vs $O(N)$)](#51-formal-mathematical-complexity-models-on2-vs-on-s-vs-on)
|
|
54
|
+
- 5.2 [Empirical Protocol Telemetry & Head-to-Head Comparison](#52-empirical-protocol-telemetry--head-to-head-comparison)
|
|
55
|
+
- 5.3 [Concrete Wire Protocol Trace Analysis: Verbatim Packet Comparison](#53-concrete-wire-protocol-trace-analysis-verbatim-packet-comparison)
|
|
56
|
+
- 5.4 [Pointer-Over-Wire Grammar & Invariant Contracts](#54-pointer-over-wire-grammar--invariant-contracts)
|
|
57
|
+
- 5.5 [Cryptographic Bundle Manifest Specification (`report.sources.json`)](#55-cryptographic-bundle-manifest-specification-reportsourcesjson)
|
|
58
|
+
6. [Synthesized Architectural Matrix & Pareto Frontier (R4)](#6-synthesized-architectural-matrix--pareto-frontier-r4)
|
|
59
|
+
- 6.1 [Exhaustive Multi-Dimensional Comparative Trade-Off Matrix](#61-exhaustive-multi-dimensional-comparative-trade-off-matrix)
|
|
60
|
+
- 6.2 [Pareto Frontier Analysis: Where Agy-Squad Dominates](#62-pareto-frontier-analysis-where-agy-squad-dominates)
|
|
61
|
+
- 6.3 [Strategic Yield Points: Where Alternative Harnesses Excel](#63-strategic-yield-points-where-alternative-harnesses-excel)
|
|
62
|
+
- 6.4 [Conclusion & Architectural Recommendations](#64-conclusion--architectural-recommendations)
|
|
63
|
+
7. [Appendices](#7-appendices)
|
|
64
|
+
- [Appendix A: Multi-Agent Protocol Tracing Harness (MAPT-Harness) Architecture](#appendix-a-multi-agent-protocol-tracing-harness-mapt-harness-architecture)
|
|
65
|
+
- [Appendix B: Seeded Defect Specifications (Event-Relay Benchmark)](#appendix-b-seeded-defect-specifications-event-relay-benchmark)
|
|
66
|
+
- [Appendix C: Telemetry Schema Specification (`telemetry.schema.json`)](#appendix-c-telemetry-schema-specification-telemetryschemajson)
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## 1. Executive Framing & The Systems Engineering Trilemma
|
|
71
|
+
|
|
72
|
+
### 1.1 The Multi-Agent Orchestration Trilemma
|
|
73
|
+
|
|
74
|
+
Building reliable, scalable, and verifiable multi-agent software engineering systems requires navigating a fundamental systems-engineering trilemma between three competing forces:
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
Orchestration Expressiveness
|
|
78
|
+
(Arbitrary Cyclic Loops, Dynamic Swarms,
|
|
79
|
+
Recursive Subgraphs, Zero-Shot Roles)
|
|
80
|
+
/\
|
|
81
|
+
/ \
|
|
82
|
+
/ \
|
|
83
|
+
/ \
|
|
84
|
+
/ THE \
|
|
85
|
+
/ TRILEMMA \
|
|
86
|
+
/ \
|
|
87
|
+
/ \
|
|
88
|
+
Context & Token Preservation <----------------> Verifiability & Determinism
|
|
89
|
+
(O(1) IPC Wire, Strict Cache (SHA-256 Digest Bundles, Host
|
|
90
|
+
Retention, Minimal Injected Bytes) Sandboxes, L41 Clean Teardown)
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
1. **Orchestration Expressiveness**: The flexibility to model arbitrary execution topologies—including non-linear cyclic graph loops, dynamic peer-to-peer runtime handoffs, map-reduce fan-outs, and zero-shot role decomposition.
|
|
94
|
+
2. **Context & Token Preservation**: The ability to execute long-horizon, multi-agent tasks without saturating LLM context windows, exhausting vendor rate limits, inducing prompt-cache evictions, or diluting model attention.
|
|
95
|
+
3. **Verifiability & Determinism**: The capacity to independently, algorithmically, and reproducibly prove that generated deliverables satisfy stated requirements, match declared inputs, maintain repository integrity, and leave zero untracked runtime pollution.
|
|
96
|
+
|
|
97
|
+
No architecture achieves maximum saturation on all three vertices simultaneously without explicit structural trade-offs. Frameworks that maximize **Orchestration Expressiveness** (e.g., AutoGen, CrewAI) sacrifice **Context Preservation** through uncontrolled message broadcast and forfeit **Verifiability** by relying on subjective, non-deterministic LLM-on-LLM debate. Conversely, systems that achieve high **Context Preservation** and **Verifiability** (such as Agy-Squad) intentionally constrain arbitrary cyclic loops in favor of schema-validated, acyclic stage compilation.
|
|
98
|
+
|
|
99
|
+
### 1.2 Signaling vs. Storage Conflation: The Root Architectural Pathology
|
|
100
|
+
|
|
101
|
+
In classical distributed systems, the control plane (IPC signaling, synchronization barriers, heartbeats) is strictly decoupled from the data plane (blob storage, file payloads, database records). Ethernet switches do not inject complete multi-megabyte database dumps into TCP SYN packets.
|
|
102
|
+
|
|
103
|
+
Modern multi-agent architectures violate this decoupling. In conventional conversational frameworks:
|
|
104
|
+
- **In-Band Payload Delivery**: When a worker agent finishes an analysis, it serializes its complete markdown deliverable, code snippets, and terminal traces into the LLM chat completion payload.
|
|
105
|
+
- **Transcript Compounding**: When this payload is returned to a supervisor or peer agent, the receiving model's context window ingests the full text. In conversational group chats, this text is subsequently re-broadcast to every participant on every subsequent turn.
|
|
106
|
+
- **The Consequence**: Prompt context bloat compounds quadratically ($O(M^2 \cdot L)$) or linearly with large state dictionaries ($O(M \cdot |S|)$). As transcripts swell into hundreds of kilobytes, prompt cache hit rates plummet, latency spikes, and models suffer from severe cognitive degradation ("lost-in-the-middle"), hallucinating file paths or forgetting initial constraints.
|
|
107
|
+
|
|
108
|
+
### 1.3 The Agy-Squad Architectural Synthesis
|
|
109
|
+
|
|
110
|
+
Agy-Squad resolves this systems crisis by applying classical operating systems principles to multi-agent LLM harnesses:
|
|
111
|
+
|
|
112
|
+
```
|
|
113
|
+
+-------------------------------------------------------------------------------+
|
|
114
|
+
| AGY-SQUAD DECOUPLED ARCHITECTURE |
|
|
115
|
+
| |
|
|
116
|
+
| +--------------------+ +--------------------+ |
|
|
117
|
+
| | Parent Session | | Specialist Subagent| |
|
|
118
|
+
| | (Squad Lead) | | (Scoped Domain) | |
|
|
119
|
+
| +---------+----------+ +---------+----------+ |
|
|
120
|
+
| | | |
|
|
121
|
+
| | 1. dispatch_squad (Manifest-Driven Compilation) | |
|
|
122
|
+
| +--------------------------------------------------->+ |
|
|
123
|
+
| | Pre-allocates isolated report slot: | |
|
|
124
|
+
| | scratch/.../specialist-1/report.md | |
|
|
125
|
+
| | | |
|
|
126
|
+
| | | 2. Executes |
|
|
127
|
+
| | | Work & |
|
|
128
|
+
| | | Writes |
|
|
129
|
+
| | v Disk Slot|
|
|
130
|
+
| | +----------------------------+ |
|
|
131
|
+
| | | Out-of-Band Report Slot: | |
|
|
132
|
+
| | | scratch/agy-squad/.../ | |
|
|
133
|
+
| | | specialist-1/report.md | |
|
|
134
|
+
| | +----------------------------+ |
|
|
135
|
+
| | 3. Pointer-Over-Wire Receipt | |
|
|
136
|
+
| | (Strictly <= 500 UTF-8 Bytes): | |
|
|
137
|
+
| | "READY scratch/.../report.md — <finding>" | |
|
|
138
|
+
| |<---------------------------------------------------+ |
|
|
139
|
+
| v |
|
|
140
|
+
| +--------------------+ |
|
|
141
|
+
| | Reactive Wakeup | |
|
|
142
|
+
| | (Single Ingestion, | |
|
|
143
|
+
| | 249 Bytes Total) | |
|
|
144
|
+
| +---------+----------+ |
|
|
145
|
+
| | |
|
|
146
|
+
| | 4. bundleReports (mcp/handoffs.mjs) |
|
|
147
|
+
| | Hashes inputs + reports into SHA-256 Manifest |
|
|
148
|
+
| v |
|
|
149
|
+
| +-------------------------------------------------------------------------+ |
|
|
150
|
+
| | Immutable Cryptographic Bundle: lead/report.sources.json | |
|
|
151
|
+
| | Algorithmic Zero-Token Verification in 20.0 ms (mcp/handoffs.mjs verify)| |
|
|
152
|
+
| +-------------------------------------------------------------------------+ |
|
|
153
|
+
+-------------------------------------------------------------------------------+
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
1. **Acyclic Staged Compilation (`dispatch_squad`)**: Tasks are compiled from deterministic manifests into discrete phases (`all_l1` lead stage followed by specialist cohorts), preventing runtime agent sprawl and infinite recursion.
|
|
157
|
+
2. **Pointer-Over-Wire Protocol**: The IPC signaling channel is restricted to single-line UTF-8 receipts capped at $\le 500$ bytes (`READY`, `PARTIAL`, `NO_REPORT`). Heavy evidentiary deliverables are written out-of-band to pre-allocated filesystem slots (`scratch/agy-squad/.../report.md`).
|
|
158
|
+
3. **Cryptographic Bundle Model**: Algorithmic SHA-256 integrity verification links verified input sources, specialist findings, and lead synthesis into an immutable JSON manifest (`report.sources.json`), validated in $< 25\text{ ms}$ at zero LLM token cost.
|
|
159
|
+
4. **Transparent Context Governance (RTK & ACS)**: Host-level pre-tool interception via Rust Token Killer (`rtk 0.51.0`) and Agy-Context-Saver (`v1.8.1+`) strips passing test noise and enforces 3-tier bounded codebase viewing.
|
|
160
|
+
5. **WSL2 Substrate & Lifecycle Hygiene (`L41`)**: Sandboxed CLI execution with deny-by-default permission profiles protecting manifests and session configs, backed by an anti-orphan PTY daemon and guaranteed teardown leaving zero untracked Git files.
|
|
161
|
+
|
|
162
|
+
---
|
|
163
|
+
|
|
164
|
+
## 2. Audited Target Group 1: General Workflow & Graph Multi-Agent Harnesses
|
|
165
|
+
|
|
166
|
+
### 2.1 MetaGPT: Standard Operating Procedure (SOP) Driven Waterfall
|
|
167
|
+
|
|
168
|
+
```
|
|
169
|
+
+------------------------------------------------------+
|
|
170
|
+
| MetaGPT Environment Message Pool |
|
|
171
|
+
+------------------------------------------------------+
|
|
172
|
+
^ | ^ |
|
|
173
|
+
1. Publish PRD | | 2. Observe PRD | |
|
|
174
|
+
| v | |
|
|
175
|
+
+----------------+ +----------------+ | |
|
|
176
|
+
| ProductManager | | Architect |-----+ 3. Publish |
|
|
177
|
+
| (Role) | | (Role) | System Spec |
|
|
178
|
+
+----------------+ +----------------+ v
|
|
179
|
+
+----------------+
|
|
180
|
+
| Engineer |
|
|
181
|
+
| (Role) |
|
|
182
|
+
+----------------+
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
#### 2.1.1 Operating Paradigm & Role Abstraction
|
|
186
|
+
MetaGPT (Hong et al., DeepWisdom, 2023) adapts traditional software engineering Standard Operating Procedures (SOPs) to multi-agent teams. Recognizing that unconstrained agent chatter produces chaotic hallucination loops, MetaGPT decomposes engineering workflows into predefined sequential stages executed by specialized role profiles (`ProductManager`, `Architect`, `ProjectManager`, `Engineer`, `QaEngineer`).
|
|
187
|
+
|
|
188
|
+
Coordination is orchestrated via a centralized `Environment` message pool. Roles register subscriptions to specific action outputs via `self._watch([ActionType])`. When an agent calls `observe()`, it filters the message pool for new events published by actions it monitors. Deliverables are constrained by requiring actions to return structured Pydantic schemas (PRD documents, Mermaid system diagrams, data structures, and file lists).
|
|
189
|
+
|
|
190
|
+
#### 2.1.2 Wire Representation & Context Degradation
|
|
191
|
+
While messages within `Environment.memory` are represented as structured `Message` objects, when an agent executes an action (e.g., `WriteDesign.run()`), it extracts the full text of upstream documents (e.g., the complete PRD, typically 1,500–4,000 tokens) and injects it verbatim into the system prompt template.
|
|
192
|
+
|
|
193
|
+
- **Wire Overhead**: Passing documents by value in memory serializes 50 KB–300 KB of text across roles in a standard project.
|
|
194
|
+
- **Token Complexity**: For $K$ sequential roles with average artifact size $L$, cumulative input token consumption scales as $O(K \cdot L)$. While linear with respect to pipeline stages, there is zero in-band pointer de-duplication: if five downstream engineers inspect the System Design, five identical copies are injected into their respective prompt contexts.
|
|
195
|
+
- **Context Exhaustion**: When upstream specifications or codebases expand, MetaGPT lacks dynamic pointer dereferencing or automated summarization pipelines. It relies on standard truncation or model window expansion.
|
|
196
|
+
|
|
197
|
+
#### 2.1.3 Failure Isolation & Execution Safety
|
|
198
|
+
- **Cascade of Upstream Errors (GIGO)**: If the `ProductManager` hallucinates an impossible dependency or the `Architect` specifies an inconsistent API signature, downstream agents ingest the error unquestioningly. Downstream roles have no native mechanism to reject upstream artifacts; they attempt to implement the flawed specification.
|
|
199
|
+
- **No Pipeline Rollback**: While local action-level retries handle Pydantic parsing failures, published messages in `Environment.memory` are immutable. There is no mechanism to rewind execution to a pre-corrupt checkpoint.
|
|
200
|
+
- **Execution Safety**: Shell commands and unit tests generated by the `Engineer` or `QaEngineer` execute directly on the host operating system via Python `subprocess.run()`. Sandboxing is absent by default; arbitrary generated code runs with the full privileges of the host user.
|
|
201
|
+
|
|
202
|
+
---
|
|
203
|
+
|
|
204
|
+
### 2.2 CrewAI: Role-Playing Collectives & Chained Delegation
|
|
205
|
+
|
|
206
|
+
```
|
|
207
|
+
+-----------------------+
|
|
208
|
+
| Crew Manager |
|
|
209
|
+
| (Process.hierarchical)|
|
|
210
|
+
+-----------------------+
|
|
211
|
+
| ^ | ^
|
|
212
|
+
Delegate Work to | | | | Delegate Work to
|
|
213
|
+
Researcher | Result| | | Writer
|
|
214
|
+
v | v |
|
|
215
|
+
+------------+ +------------+
|
|
216
|
+
| Researcher | | Writer |
|
|
217
|
+
| (Agent) | | (Agent) |
|
|
218
|
+
+------------+ +------------+
|
|
219
|
+
| |
|
|
220
|
+
v v
|
|
221
|
+
[Search Tool] [File Tool]
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
#### 2.2.1 Operating Paradigm (Sequential vs. Hierarchical)
|
|
225
|
+
CrewAI (João Moura, 2023–2024) models agent interactions after human corporate teams using three primary primitives:
|
|
226
|
+
- **`Agent`**: Encapsulates a persona with `role`, `goal`, `backstory`, optional `llm`, `tools`, and execution constraints (`max_iter`).
|
|
227
|
+
- **`Task`**: Defines a work unit with `description`, `expected_output`, assigned agent, and `context` (references to prerequisite tasks).
|
|
228
|
+
- **`Crew`**: Executes tasks using either `Process.sequential` (linear task execution array) or `Process.hierarchical` (a centralized manager agent dynamically delegates tasks to coworkers via tool calls).
|
|
229
|
+
|
|
230
|
+
#### 2.2.2 Wire Representation & Token Overhead
|
|
231
|
+
- **Sequential String Injection**: In `Process.sequential`, task handoffs occur via raw string concatenation. If Task $B$ declares `context=[TaskA]`, the entire raw string output of Task $A$ is injected into Task $B$'s prompt under: `This is the context you're working with: <task_a_output>`.
|
|
232
|
+
- **Hierarchical Tool Churn**: In `Process.hierarchical`, the manager coordinates coworkers via LangChain-style tool calling (`Delegate work to coworker`). The coworker executes and returns its full output string into the manager's conversational scratchpad.
|
|
233
|
+
- **Token Inflation**: In multi-task chains, context expands aggressively ($O(K^2 \cdot T)$). A 5-task crew frequently generates 40,000–100,000 cumulative LLM tokens to produce a 500-word final deliverable.
|
|
234
|
+
- **Vector Memory vs. Prompt Bloat**: Although CrewAI integrates ChromaDB (short-term) and SQLite (long-term) memory, active task execution prompts bypass vector retrieval for task prerequisites, relying on raw string injection that rapidly exhausts context windows.
|
|
235
|
+
|
|
236
|
+
#### 2.2.3 Failure Modes & Blast Radius
|
|
237
|
+
- **Manager Context Contamination**: In hierarchical mode, if a coworker returns invalid or contradictory data, the manager agent frequently enters a repetitive loop (calling `Ask question to coworker` repeatedly until exhausting `max_iter`), consuming token budgets without making forward progress.
|
|
238
|
+
- **Shared Memory Blast Radius**: All agents execute within the same Python process memory space. A crash or state corruption in a custom tool halts the entire crew.
|
|
239
|
+
- **Execution Safety**: Tools run directly in the host Python environment without containerization or permission gating.
|
|
240
|
+
|
|
241
|
+
---
|
|
242
|
+
|
|
243
|
+
### 2.3 AutoGen & OpenAI Swarm: Conversational Broadcast & Stateless Swarms
|
|
244
|
+
|
|
245
|
+
```
|
|
246
|
+
+-----------------------------------------------------------+
|
|
247
|
+
| AutoGen GroupChat Broadcast |
|
|
248
|
+
+-----------------------------------------------------------+
|
|
249
|
+
| | |
|
|
250
|
+
v (Full History) v (Full History) v (Full History)
|
|
251
|
+
+---------------+ +---------------+ +---------------+
|
|
252
|
+
| UserProxy | | CoderAgent | | CriticAgent |
|
|
253
|
+
+---------------+ +---------------+ +---------------+
|
|
254
|
+
| | |
|
|
255
|
+
+----------------------+----------------------+
|
|
256
|
+
|
|
|
257
|
+
Broadcasts New Message to ALL
|
|
258
|
+
|
|
259
|
+
-------------------------------------------------------------------------
|
|
260
|
+
|
|
261
|
+
+-------------------------------------------+
|
|
262
|
+
| OpenAI Swarm Function Handoff |
|
|
263
|
+
+-------------------------------------------+
|
|
264
|
+
Agent A receives (messages[])
|
|
265
|
+
|
|
|
266
|
+
| Calls transfer_to_agent_b()
|
|
267
|
+
v
|
|
268
|
+
Agent B receives (messages[] + tool_calls + handoff)
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
#### 2.3.1 Operating Paradigm: Broadcast vs. Function Handoffs
|
|
272
|
+
- **AutoGen (Microsoft; Wu et al., 2023)**: Organizes collaboration around conversational dialogues between `ConversableAgent` instances. In multi-agent settings, `GroupChat` and `GroupChatManager` manage conversations among $N$ agents. When an agent speaks, its message is **broadcast to all participants**, each maintaining a local transcript of the entire group history.
|
|
273
|
+
- **OpenAI Swarm (OpenAI Experimental, 2024)**: Implements client-side multi-agent coordination centered on **stateless function handoffs**. When an agent invokes a handoff function returning another `Agent` instance (`return agent_b`), the client loop switches execution to Agent B, passing the accumulated `messages` array forward.
|
|
274
|
+
|
|
275
|
+
#### 2.3.2 Wire Representation & $O(N^2)$ Quadratic Token Inflation
|
|
276
|
+
Both frameworks pass complete conversational transcripts on every API turn. The wire payload consists of the accumulated JSON array of chat completion messages:
|
|
277
|
+
$$\text{Tokens}_{\text{Broadcast}} = N \sum_{m=1}^{M} (m \cdot L) = \frac{N \cdot M(M+1)}{2} L = O(N \cdot M^2 \cdot L)$$
|
|
278
|
+
For a 5-agent group chat exchanging 30 turns with an average message length of 500 tokens, prompt token consumption exceeds 225,000 tokens for minimal semantic output. In OpenAI Swarm, the linear message array expands monotonically, passing the complete historical chain of previous tool executions to each new agent.
|
|
279
|
+
|
|
280
|
+
#### 2.3.3 Failure Modes & Execution Boundaries
|
|
281
|
+
- **Polite Degeneration Loops**: In AutoGen `GroupChat`, agents frequently enter circular conversational loops ("Thank you!", "You're welcome! Let me know if you need anything else!"), burning token budgets until hitting `max_consecutive_auto_reply`.
|
|
282
|
+
- **Hallucination Cascades**: The shared conversational transcript is a single failure domain. A single hallucinated premise broadcast to the group is accepted by peer agents and compounded in subsequent turns.
|
|
283
|
+
- **Execution Sandboxing**: AutoGen provides the strongest sandbox among general workflow frameworks via its `DockerCommandLineCodeExecutor`, which executes agent code inside isolated Docker containers. OpenAI Swarm provides zero sandboxing, running functions on the host Python process.
|
|
284
|
+
|
|
285
|
+
---
|
|
286
|
+
|
|
287
|
+
### 2.4 LangGraph: Statecharts, Controlled Cyclic Graphs, & Channel Reducers
|
|
288
|
+
|
|
289
|
+
```
|
|
290
|
+
+-------------------+
|
|
291
|
+
| Start Node |
|
|
292
|
+
+-------------------+
|
|
293
|
+
|
|
|
294
|
+
v
|
|
295
|
+
+-------------------+
|
|
296
|
+
+------------->| Supervisor |-------------+
|
|
297
|
+
| | (Router Node) | |
|
|
298
|
+
| +-------------------+ |
|
|
299
|
+
| / \ |
|
|
300
|
+
| route=="math"/ \route=="code" |
|
|
301
|
+
| v v |
|
|
302
|
+
| +-------------+ +-------------+ |
|
|
303
|
+
| | Mathematic | | Programmer | |
|
|
304
|
+
| | Worker | | Worker | |
|
|
305
|
+
| +-------------+ +-------------+ |
|
|
306
|
+
| \ / |
|
|
307
|
+
| +--------+-------+ |
|
|
308
|
+
| | |
|
|
309
|
+
+-------------------------+ v
|
|
310
|
+
State Updated via Reducer route=="finish"
|
|
311
|
+
(State Persistence / Checkpoint) |
|
|
312
|
+
v
|
|
313
|
+
+---------------+
|
|
314
|
+
| End Node |
|
|
315
|
+
+---------------+
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
#### 2.4.1 Operating Paradigm: StateGraph, Nodes, & Reducers
|
|
319
|
+
LangGraph (LangChain; Chase et al., 2024) models multi-agent workflows as **Statecharts and Controlled Cyclic Graphs**. It departs from conversational chat rooms by grounding coordination in explicit state transitions:
|
|
320
|
+
- **`StateGraph(state_schema)`**: Defines a centralized schema (TypedDict or Pydantic model) representing the total execution state.
|
|
321
|
+
- **Nodes**: Python functions or runnables that accept the current `state` and return discrete state updates.
|
|
322
|
+
- **Channel Reducers (`Annotated[Type, reducer]`)**: Define how node updates are merged into state channels. For example, `Annotated[list, operator.add]` appends new messages, while custom reducers can overwrite, merge dictionaries, or aggregate metrics.
|
|
323
|
+
- **Edges**: Deterministic (`add_edge`) or conditional (`add_conditional_edges`) routing functions governing state transitions.
|
|
324
|
+
|
|
325
|
+
#### 2.4.2 Wire Representation & Context Preservation
|
|
326
|
+
- **Decoupled State vs. Context Window**: LangGraph separates persistent execution state (stored in checkpointers) from the LLM prompt context window. Nodes return minimal state deltas (`{"iteration": 1}`).
|
|
327
|
+
- **Overhead Variability**: When developers use domain-specific state channels (e.g. `{"diff": str, "status": bool}`), wire overhead is minimal ($O(M \cdot S)$). However, when developers adopt the default `messages: Annotated[list, add_messages]` pattern, state channels accumulate unpruned transcripts, degenerating to $O(M^2)$ if passed directly to LLMs.
|
|
328
|
+
- **Checkpointers (`BaseCheckpointSaver`)**: Every graph super-step is saved to an external store (`MemorySaver`, `SqliteSaver`, `PostgresSaver`), enabling pause/resume, time-travel debugging, and Human-in-the-Loop (HITL) inspection barriers (`interrupt_before`, `interrupt_after`).
|
|
329
|
+
|
|
330
|
+
#### 2.4.3 Failure Isolation & Execution Safety
|
|
331
|
+
- **State Rewind & Time-Travel**: Because state is snapshotted at each super-step, developers or supervisor nodes can roll back execution to any prior `checkpoint_id`, edit the state, and resume along a corrected trajectory.
|
|
332
|
+
- **Lack of Cryptographic Attestation**: Checkpoint IDs are arbitrary UUID4s or timestamps. LangGraph does not compute SHA-256 digests over deliverables, does not generate Merkle trees, and provides no tamper-evident proof that state stored in Postgres has not been modified out-of-band.
|
|
333
|
+
- **Execution Safety**: LangGraph is purely an in-process orchestration library; it provides zero native OS sandboxing. Tools execute on the host Python runtime unless external container infrastructure is provided.
|
|
334
|
+
|
|
335
|
+
---
|
|
336
|
+
|
|
337
|
+
## 3. Audited Target Group 2: Autonomous Coding & SWE Harnesses
|
|
338
|
+
|
|
339
|
+
### 3.1 Devin / Cognition Sol: Cloud MicroVMs & Snapshot-Resume Workspace
|
|
340
|
+
|
|
341
|
+
```
|
|
342
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
343
|
+
│ Devin Control Plane │
|
|
344
|
+
│ ┌───────────────────────┐ ┌───────────────────────┐ │
|
|
345
|
+
│ │ Hierarchical Planner │ ◄───► │ Observation Filter │ │
|
|
346
|
+
│ └───────────┬───────────┘ └───────────▲───────────┘ │
|
|
347
|
+
└──────────────┼───────────────────────────────┼──────────────┘
|
|
348
|
+
│ Tool RPC (JSON-RPC) │ Filtered Events
|
|
349
|
+
▼ │
|
|
350
|
+
┌──────────────────────────────────────────────┴──────────────┐
|
|
351
|
+
│ Devbox Cloud Execution Plane (KVM) │
|
|
352
|
+
│ ┌───────────────────────────────────────────────────────┐ │
|
|
353
|
+
│ │ Linux MicroVM (Firecracker / Cloud Hypervisor) │ │
|
|
354
|
+
│ │ ┌───────────────┐ ┌───────────────┐ ┌─────────────┐ │ │
|
|
355
|
+
│ │ │ Headless Chrome│ │ Shell Runner │ │ Monaco Code │ │ │
|
|
356
|
+
│ │ │ (CDP Browser) │ │ (PTY Daemon) │ │ Editor │ │ │
|
|
357
|
+
│ │ └───────────────┘ └───────────────┘ └─────────────┘ │ │
|
|
358
|
+
│ │ ┌─────────────────────────────────────────────────┐ │ │
|
|
359
|
+
│ │ │ Snapshot & Block Storage Layer (S3 / CoW Diff) │ │ │
|
|
360
|
+
│ │ └─────────────────────────────────────────────────┘ │ │
|
|
361
|
+
│ └───────────────────────────────────────────────────────┘ │
|
|
362
|
+
└─────────────────────────────────────────────────────────────┘
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
#### 3.1.1 Decoupled Architecture: Control Plane vs. Devbox
|
|
366
|
+
Devin (Cognition AI) enforces complete architectural separation between the cognitive planning engine ("The Brain") and the runtime execution plane ("The Workspace"):
|
|
367
|
+
- **Control Plane ("The Brain")**: Cloud-hosted, stateless planning service that maintains roadmaps, decomposes user specifications, and emits tool RPC calls.
|
|
368
|
+
- **Execution Plane ("Devbox")**: Dedicated remote Linux microVM executing shell commands, maintaining filesystems, compiling code, and rendering frontend web apps.
|
|
369
|
+
- **Observation Filtering**: Command stdout/stderr and file modifications are streamed out-of-band to an observation aggregator that summarizes terminal output before forwarding it to the planning model's prompt context.
|
|
370
|
+
|
|
371
|
+
#### 3.1.2 Hardware-Level Virtualization & Blast Radius Isolation
|
|
372
|
+
Rather than relying on shared-kernel Docker containers, Devin isolates workspaces within dedicated Linux KVM microVMs (e.g., AWS Firecracker).
|
|
373
|
+
- **Kernel-Level Confinement**: Each session runs its own guest Linux kernel, virtualized network stack, and block devices. Fork-bombs, malicious dependencies, or privilege escalations are confined strictly to the guest VM.
|
|
374
|
+
- **Tri-Modal Modalities**: Integrates a persistent pseudo-terminal daemon (PTY), a Monaco-based structured code editor manipulating source trees via Language Server Protocol (LSP), and a headless Chromium browser controlled via Chrome DevTools Protocol (CDP) for DOM inspection and visual regression testing.
|
|
375
|
+
- **Copy-on-Write (CoW) Snapshots**: The underlying microVM filesystem is backed by snapshot-capable block storage. Before risky operations (refactoring, major dependency updates), Devin snapshots the machine state. If an approach fails, the VM reverts to the pre-task snapshot in sub-second latency.
|
|
376
|
+
|
|
377
|
+
#### 3.1.3 Strategic Yield Points & Limitations
|
|
378
|
+
- **Cloud Lock-In & Opacity**: Closed-source proprietary SaaS that cannot run offline or in air-gapped secure enclaves.
|
|
379
|
+
- **High Infrastructure Latency & Cost**: Booting, snapshotting, and provisioning cloud microVMs incurs substantial financial overhead and network round-trip latency.
|
|
380
|
+
- **Absence of Cryptographic Attestation**: Verification relies on subjective LLM evaluation and unit test execution; Devin does not produce tamper-evident SHA-256 cryptographic manifests linking source inputs to verified deliverables.
|
|
381
|
+
|
|
382
|
+
---
|
|
383
|
+
|
|
384
|
+
### 3.2 Claude Code & SWE-agent: The Agent-Computer Interface (ACI)
|
|
385
|
+
|
|
386
|
+
```
|
|
387
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
388
|
+
│ Claude Code CLI Client │
|
|
389
|
+
│ ┌───────────────────────┐ ┌───────────────────────┐ │
|
|
390
|
+
│ │ Ink Terminal UI / REPL│ ◄───► │ Context Manager │ │
|
|
391
|
+
│ │ (User Approval Gate) │ │ (/context, /compact) │ │
|
|
392
|
+
│ └───────────┬───────────┘ └───────────▲───────────┘ │
|
|
393
|
+
└──────────────┼───────────────────────────────┼──────────────┘
|
|
394
|
+
│ Tool Requests │ Output / Diffs
|
|
395
|
+
▼ │
|
|
396
|
+
┌──────────────────────────────────────────────┴──────────────┐
|
|
397
|
+
│ Local Execution Plane │
|
|
398
|
+
│ ┌───────────────────────────────────────────────────────┐ │
|
|
399
|
+
│ │ Host Shell (bash/zsh) / Optional Docker Container │ │
|
|
400
|
+
│ │ ┌───────────────┐ ┌───────────────┐ ┌─────────────┐ │ │
|
|
401
|
+
│ │ │ Git CLI / Diff│ │ File System │ │ Test Suites │ │ │
|
|
402
|
+
│ │ │ Tooling │ │ (Direct I/O) │ │ Runner │ │ │
|
|
403
|
+
│ │ └───────────────┘ └───────────────┘ └─────────────┘ │ │
|
|
404
|
+
│ └───────────────────────────────────────────────────────┘ │
|
|
405
|
+
└─────────────────────────────────────────────────────────────┘
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
#### 3.2.1 Princeton SWE-agent & ACI Principles
|
|
409
|
+
SWE-agent (Princeton NLP; Yang et al., 2024) pioneered the **Agent-Computer Interface (ACI)**, establishing that standard Unix shell interfaces designed for human visual scrolling are hostile to LLMs. Unbounded tool output quickly overflows context windows. The ACI enforces:
|
|
410
|
+
- **Bounded Viewports**: File inspection is strictly windowed (e.g., displaying exactly 100 lines at a time with line numbers).
|
|
411
|
+
- **Diff-Based Editing**: Replaces whole-file rewriting with structured search-and-replace edits backed by immediate linter feedback.
|
|
412
|
+
- **SWE-bench Adapter**: Connects agents to Docker-isolated conda environments hosting benchmark repositories, capturing git patches upon completion.
|
|
413
|
+
|
|
414
|
+
#### 3.2.2 Claude Code CLI Architecture & Context Compaction
|
|
415
|
+
Anthropic's Claude Code translates ACI principles into a production terminal harness:
|
|
416
|
+
- **Host Execution Harness**: Built on Node.js/TypeScript using an Ink-based terminal UI (React in terminal). It runs directly on macOS, Linux, or WSL2.
|
|
417
|
+
- **Targeted Diff Editing & JIT Loading**: Minimizes prompt token consumption by reading file slices via `GlobTool`/`GrepTool` and applying localized diffs.
|
|
418
|
+
- **Context Compaction Lifecycle (`/compact`)**: When context usage approaches ~95% capacity, Claude Code initiates lossy compaction:
|
|
419
|
+
1. The multi-turn conversation history is summarized via an LLM prompt.
|
|
420
|
+
2. Intermediate tool stdout, discarded code traces, and redundant file contents are pruned.
|
|
421
|
+
3. A consolidated state summary is emitted, preserving anchor files (`CLAUDE.md`) while discarding fine-grained execution history.
|
|
422
|
+
- **Execution Hazards**: Runs directly on the developer's workstation by default. While bash commands require interactive user approval, human oversight is prone to fatigue; approving an unintended command risks destructive filesystem mutations.
|
|
423
|
+
|
|
424
|
+
---
|
|
425
|
+
|
|
426
|
+
### 3.3 Google Antigravity Native `/teamwork`: Hierarchical Swarms & Reactive Wakeups
|
|
427
|
+
|
|
428
|
+
```
|
|
429
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
430
|
+
│ Google Antigravity /teamwork │
|
|
431
|
+
│ │
|
|
432
|
+
│ ┌───────────────────────┐ │
|
|
433
|
+
│ │ teamwork_orchestrator │ │
|
|
434
|
+
│ └───────────┬───────────┘ │
|
|
435
|
+
│ ┌───────────────────┼───────────────────┐ │
|
|
436
|
+
│ ▼ ▼ ▼ │
|
|
437
|
+
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
|
438
|
+
│ │ explorer_1 │ │ explorer_2 │ │ miner_1 │ │
|
|
439
|
+
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
|
|
440
|
+
│ └───────────────────┼───────────────────┘ │
|
|
441
|
+
│ ▼ │
|
|
442
|
+
│ ┌─────────────────┐ │
|
|
443
|
+
│ │ worker_report_1 │ │
|
|
444
|
+
│ └────────┬────────┘ │
|
|
445
|
+
│ ┌───────────────────┴───────────────────┐ │
|
|
446
|
+
│ ▼ ▼ │
|
|
447
|
+
│ ┌─────────────┐ ┌─────────────┐ │
|
|
448
|
+
│ │ reviewer_1 │ │ auditor_1 │ │
|
|
449
|
+
│ └─────────────┘ └─────────────┘ │
|
|
450
|
+
└─────────────────────────────────────────────────────────────┘
|
|
451
|
+
```
|
|
452
|
+
|
|
453
|
+
#### 3.3.1 Hierarchical Swarm Topology & Metadata Conventions
|
|
454
|
+
Google Antigravity's native `/teamwork` engine coordinates multi-agent engineering workflows through a structured hierarchy:
|
|
455
|
+
- **Top-Level Orchestrator (`teamwork_orchestrator`)**: Plans milestones (`plan.md`) and dynamically spawns subagents.
|
|
456
|
+
- **Domain Explorers / Workers**: Parallel subagents investigating codebases, modifying files, or executing tests.
|
|
457
|
+
- **Dedicated Report Writer (`worker_report_1`)**: Aggregates disparate worker findings into unified deliverables.
|
|
458
|
+
- **Review & Audit Fleet (`reviewer_1`, `auditor_1`)**: Independent verification subagents deployed to perform secondary semantic reviews of code and documentation.
|
|
459
|
+
- **Metadata Conventions (`.agents/teamwork/`)**: Persistent coordination state is tracked via filesystem artifacts:
|
|
460
|
+
- `BRIEFING.md`: Working memory of each agent (~100 lines), with append-only (🔒) identity and constraints.
|
|
461
|
+
- `progress.md`: Liveness heartbeat updated every step to enable watchdog dead-agent detection.
|
|
462
|
+
- `handoff.md`: 5-Component handoff report (`Observation`, `Logic Chain`, `Caveats`, `Conclusion`, `Verification Method`).
|
|
463
|
+
|
|
464
|
+
#### 3.3.2 Event Engine: Reactive Wakeup & Watchdog Schedules
|
|
465
|
+
- **Reactive Wakeups**: Subagents execute asynchronously without busy-polling. When a subagent finishes or sends a message, Antigravity injects a `<SYSTEM_MESSAGE>` that reactively resumes the parent orchestrator.
|
|
466
|
+
- **Coordinated Watchdog Timers**: For operations exceeding 15 seconds, `/teamwork` schedules watchdog timers via `schedule(DurationSeconds=45, Prompt="...", TimerCondition="task-X")`.
|
|
467
|
+
|
|
468
|
+
#### 3.3.3 Architectural Pathologies & Cost Drivers
|
|
469
|
+
Head-to-head empirical auditing (`benchmarks/teamwork-vs-squad/`, `docs/benchmarks.md`) on the canonical 6-defect event-relay fixture reveals three severe architectural inefficiencies in native `/teamwork`:
|
|
470
|
+
1. **Conversational Wakeup Spam & Context Flooding**: Rather than sending compact receipts, `/teamwork` streams multi-kilobyte intermediate updates directly into parent prompt contexts. In the 6-defect benchmark, `/teamwork` injected **4,940 bytes across 5 wakeups** into the parent prompt (vs. 249 bytes in Agy-Squad).
|
|
471
|
+
2. **Review Fleet Transcript Overhead**: To verify reports, `/teamwork` dispatched two full subagents (`reviewer_1` and `auditor_1`) that consumed **174 steps and 228.6 KB of transcript** re-reading files and executing redundant semantic reviews.
|
|
472
|
+
3. **Workspace File Abandonment & Lifecycle Leakage**: Lacking an automated post-run cleanup contract, `/teamwork` abandoned **41 untracked files totaling 207.9 KB** across `.agents/teamwork/` (`Finding F2`).
|
|
473
|
+
|
|
474
|
+
---
|
|
475
|
+
|
|
476
|
+
### 3.4 Agy-Squad Reference Baseline: Staged Manifests & Cryptographic Governance
|
|
477
|
+
|
|
478
|
+
```
|
|
479
|
+
┌─────────────────────────────────────────────────────────────────────────┐
|
|
480
|
+
│ Agy-Squad Governed Architecture │
|
|
481
|
+
│ │
|
|
482
|
+
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
|
483
|
+
│ │ Staged Manifest Compiler │ │
|
|
484
|
+
│ │ manifest.json ───► dispatch_squad (8.5 ms compile) │ │
|
|
485
|
+
│ │ Reserves exact report slots: scratch/agy-squad/.../report.md │ │
|
|
486
|
+
│ └─────────────────────────────────┬─────────────────────────────────┘ │
|
|
487
|
+
│ │ │
|
|
488
|
+
│ ┌──────────────────────────┴──────────────────────────┐ │
|
|
489
|
+
│ ▼ ▼ │
|
|
490
|
+
│ ┌───────────────────────────────┐ ┌───────────────────────┐ │
|
|
491
|
+
│ │ Squad Lead (all_l1) │ │ Domain Specialists │ │
|
|
492
|
+
│ │ Direct Code Verification │ │ (ab_audit) │ │
|
|
493
|
+
│ └──────────────┬────────────────┘ └───────────┬───────────┘ │
|
|
494
|
+
│ │ │ │
|
|
495
|
+
│ │ Single-Line Receipts (<= 500 B) │ │
|
|
496
|
+
│ │ "READY scratch/.../report.md — obs" │ │
|
|
497
|
+
│ ▼ ▼ │
|
|
498
|
+
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
|
499
|
+
│ │ mcp/handoffs.mjs Cryptographic Engine │ │
|
|
500
|
+
│ │ - Atomic publishing (openSync wx, fsync, linkSync) │ │
|
|
501
|
+
│ │ - SHA-256 Bundle generation: squad-ab_audit.sources.json │ │
|
|
502
|
+
│ │ - Out-of-band digest verification in 20.0 ms (Zero Tokens!) │ │
|
|
503
|
+
│ └─────────────────────────────────┬─────────────────────────────────┘ │
|
|
504
|
+
│ │ │
|
|
505
|
+
│ ┌─────────────────────────────────┴─────────────────────────────────┐ │
|
|
506
|
+
│ │ Context Governance & OS Sandboxing Substrate │ │
|
|
507
|
+
│ │ - RTK (Rust Token Killer): Transparent shell rewriting │ │
|
|
508
|
+
│ │ - 3-Tier Codebase Inspection: waymark_ask -> view_file -> rtk │ │
|
|
509
|
+
│ │ - WSL2 / POSIX Sandbox settings: Deny-by-default write policies │ │
|
|
510
|
+
│ │ - L41 Teardown Contract: 0 untracked Git files post-execution │ │
|
|
511
|
+
│ └───────────────────────────────────────────────────────────────────┘ │
|
|
512
|
+
└─────────────────────────────────────────────────────────────────────────┘
|
|
513
|
+
```
|
|
514
|
+
|
|
515
|
+
#### 3.4.1 Staged Compiler Pipeline (`mcp/dispatch.mjs`)
|
|
516
|
+
Agy-Squad disallows monolithic swarm dispatch. Orchestration is compiled into discrete, acyclic stages:
|
|
517
|
+
1. **Lead Stage (`squadId: "all_l1"`)**: Compiles exactly one Squad Lead per squad defined in `manifest.json`. The lead's prompt contains supervisor cohort rules (`superviseCohort`), authorizing it to call `dispatch_squad` once with its specific squad ID.
|
|
518
|
+
2. **Specialist Stage (`squadId: "<id>"`)**: Compiles 1 to $N$ domain specialists. The compiler pre-allocates collision-free report slots on disk:
|
|
519
|
+
`scratch/agy-squad/run-<uuid>/plan-<hex>/squad-<id>/specialist-<N>/report.md`
|
|
520
|
+
3. **Symlink TOCTOU Hardening**: `readManifest` validates that candidate manifests remain strictly inside the workspace root using `fs.realpathSync`, `O_NOFOLLOW | O_NONBLOCK`, and Linux `/proc/self/fd/${fd}` verification.
|
|
521
|
+
|
|
522
|
+
#### 3.4.2 Pointer-Over-Wire Protocol (`mcp/handoffs.mjs`)
|
|
523
|
+
The IPC signaling wire is strictly decoupled from evidentiary storage:
|
|
524
|
+
- **Receipt Ceiling**: Inter-agent messages are capped at $\le 500$ UTF-8 bytes and restricted to a single line (`/[\r\n\u2028\u2029]/` triggers immediate rejection).
|
|
525
|
+
- **Discrete Statuses**:
|
|
526
|
+
- `READY <assigned-slot> — <concrete observation and implication>`
|
|
527
|
+
- `PARTIAL <assigned-slot> — <blocker description>`
|
|
528
|
+
- `NO_REPORT — <fatal initialization reason>`
|
|
529
|
+
- **Handoff Reconciliation**: `reconcileHandoffs` verifies that incoming messages map 1-to-1 to recorded subagent `conversationId`s, point to pre-allocated report slots, and contain zero conflicting receipts.
|
|
530
|
+
|
|
531
|
+
#### 3.4.3 Cryptographic Bundle Model & Zero-Token Verification
|
|
532
|
+
Verification is partitioned into semantic inspection and algorithmic integrity:
|
|
533
|
+
1. **Semantic Inspection**: The Squad Lead reads cited lines and runs test commands (`view_file`, `npm test`).
|
|
534
|
+
2. **Cryptographic Bundling (`mcp/handoffs.mjs bundle`)**:
|
|
535
|
+
- Re-hashes primary source files (`verifiedFiles: { "gateway.mjs": "<sha256>" }`).
|
|
536
|
+
- Parses the lead's synthesis markdown for an unambiguous `PASS` verdict (stripping code blocks and detecting negations).
|
|
537
|
+
- Archives all specialist markdown reports and their SHA-256 digests into an immutable JSON bundle: `lead/report.sources.json`.
|
|
538
|
+
3. **Out-of-Band Verification (`mcp/handoffs.mjs verify`)**: Validates the complete bundle in **$20.0\text{ ms}$ at zero LLM token cost**. If any source code, report content, or synthesis conclusion has drifted by 1 byte, verification throws an immediate fatal error.
|
|
539
|
+
|
|
540
|
+
#### 3.4.4 RTK & 3-Tier Codebase Inspection
|
|
541
|
+
- **Rust Token Killer (`rtk 0.51.0`)**: Pre-tool hooks intercept shell commands, routing `npm test`, `pytest`, `cargo test`, and `git diff` through RTK binaries (`rtk test`, `rtk err`, `rtk summary`). Passing boilerplate is stripped; only actionable errors are emitted (saving **839.3K tokens** / $31.4\%$ in benchmark tests).
|
|
542
|
+
- **3-Tier Inspection Hierarchy**:
|
|
543
|
+
- *Tier 1 (Structural Coordinates)*: `waymark_ask(plain: true)` discovers exact AST coordinates (~19–466 tokens).
|
|
544
|
+
- *Tier 2 (Precision Slices)*: `view_file` reads bounded 1–150 line slices.
|
|
545
|
+
- *Tier 3 (Compact Exploration)*: `rtk read`, `rtk find`, `rtk grep` / `rtk rg`.
|
|
546
|
+
|
|
547
|
+
#### 3.4.5 WSL2 Substrate, Permissions Confinement, and `L41` Teardown
|
|
548
|
+
- **Permissions Sandbox (`bin/agy-squad.mjs`)**: CLI sessions launch with explicit settings:
|
|
549
|
+
```json
|
|
550
|
+
{
|
|
551
|
+
"trustedWorkspaces": ["/workspace"],
|
|
552
|
+
"allowNonWorkspaceAccess": false,
|
|
553
|
+
"permissions": {
|
|
554
|
+
"allow": ["read_file(/workspace)", "mcp(dispatch_squad)", "write_file(/run)"],
|
|
555
|
+
"deny": ["write_file(/plugin)", "write_file(/agent.md)", "write_file(/manifest.json)"]
|
|
556
|
+
}
|
|
557
|
+
}
|
|
558
|
+
```
|
|
559
|
+
Self-modification attacks are impossible: subagents cannot alter their own manifests, plugin registrations, or agent profiles.
|
|
560
|
+
- **Governed 3-Scope Teardown Contract (`L41`)**: Upon completion, `pruneReports` deletes intermediate specialist markdown files while preserving the immutable `report.sources.json` and lead synthesis. Temporary agent profiles and ephemeral MCP symlinks are unlinked, guaranteeing **`0` untracked Git files** remaining in the repository.
|
|
561
|
+
|
|
562
|
+
---
|
|
563
|
+
|
|
564
|
+
## 4. 5-Axis Quantitative Evaluation Rubric & Comparative Scores
|
|
565
|
+
|
|
566
|
+
### 4.1 Rubric Metric Definitions & 1.0–5.0 Scale Anchors
|
|
567
|
+
|
|
568
|
+
To evaluate multi-agent harnesses objectively, we establish a standardized 5-axis scoring rubric on a 1.0 to 5.0 scale:
|
|
569
|
+
|
|
570
|
+
```
|
|
571
|
+
┌────────────────────────────────────────────────────────────────────────┐
|
|
572
|
+
│ 5-Axis Evaluation Rubric Anchors │
|
|
573
|
+
├───────────────────────┬────────────────────────────────────────────────┤
|
|
574
|
+
│ Evaluation Axis │ Anchor Definitions (1.0 vs 3.0 vs 5.0) │
|
|
575
|
+
├───────────────────────┼────────────────────────────────────────────────┤
|
|
576
|
+
│ 1. Token & Context │ 1.0: Full-transcript broadcast; raw logs dumped│
|
|
577
|
+
│ Preservation │ 3.0: Lossy summarization; naive head/tail cut │
|
|
578
|
+
│ (TCP) │ 5.0: <=500B receipts; out-of-band slots; RTK │
|
|
579
|
+
├───────────────────────┼────────────────────────────────────────────────┤
|
|
580
|
+
│ 2. Evidence │ 1.0: Ephemeral chat promises; "TERMINATE" regex│
|
|
581
|
+
│ Traceability & │ 3.0: Git commits & test exit codes in transcript│
|
|
582
|
+
│ Crypto Proof (ETC) │ 5.0: SHA-256 digest bundles; sub-25ms verify │
|
|
583
|
+
├───────────────────────┼────────────────────────────────────────────────┤
|
|
584
|
+
│ 3. Failure Isolation &│ 1.0: Single failure domain; chat cascade loops │
|
|
585
|
+
│ Blast Radius │ 3.0: Sub-task branches; manual retry triggers │
|
|
586
|
+
│ (FIB) │ 5.0: Pre-allocated slots; schema reconcile; CoW│
|
|
587
|
+
├───────────────────────┼────────────────────────────────────────────────┤
|
|
588
|
+
│ 4. Orchestration │ 1.0: Rigid linear pipelines; no branching │
|
|
589
|
+
│ Topology │ 3.0: Dynamic unconstrained swarm sprawl │
|
|
590
|
+
│ Flexibility (OTF) │ 5.0: Arbitrary cyclic statecharts; staged DAGs │
|
|
591
|
+
├───────────────────────┼────────────────────────────────────────────────┤
|
|
592
|
+
│ 5. OS & Execution │ 1.0: Arbitrary bare host execution; no sandbox │
|
|
593
|
+
│ Boundary Safety │ 3.0: Docker containerization with volume mount │
|
|
594
|
+
│ (OES) │ 5.0: KVM microVMs / deny-by-default permissions│
|
|
595
|
+
└───────────────────────┴────────────────────────────────────────────────┘
|
|
596
|
+
```
|
|
597
|
+
|
|
598
|
+
1. **Token & Context Preservation (TCP)**: Ability to minimize communication payload size, eliminate transcript bloat, preserve prompt KV cache prefixes, and prevent context saturation.
|
|
599
|
+
- *1.0 (Pathological)*: Full-transcript conversational broadcast ($O(N^2)$); raw command logs dumped into prompt ($> 15\text{ KB}$/turn).
|
|
600
|
+
- *3.0 (Moderate)*: Lossy summarization triggered at ~95% context capacity; windowed tool viewing.
|
|
601
|
+
- *5.0 (Optimal)*: Strict $\le 500\text{ B}$ pointer receipt ceiling; out-of-band payload delivery; transparent AST/regex command filtering (RTK); deterministic cache retention.
|
|
602
|
+
2. **Evidence Traceability & Cryptographic Proof (ETC)**: Availability of tamper-evident proofs, content-addressed deliverables, and reproducible algorithmic verification.
|
|
603
|
+
- *1.0 (Pathological)*: Ephemeral chat promises; no file artifacts; regex completion checks (`"TERMINATE"`).
|
|
604
|
+
- *3.0 (Moderate)*: Git commits and diff patches stored in repository; test execution logs recorded in chat history.
|
|
605
|
+
- *5.0 (Optimal)*: Tamper-evident SHA-256 digest bundles covering all source inputs, specialist reports, and lead synthesis; sub-25ms algorithmic verification; zero-token artifact integrity.
|
|
606
|
+
3. **Failure Isolation & Blast Radius (FIB)**: Prevention of error cascades, containment of semantic hallucinations, and state rollback capability.
|
|
607
|
+
- *1.0 (Pathological)*: Shared global prompt memory; a hallucination or corrupt payload cascades to all agents; unisolated host shell.
|
|
608
|
+
- *3.0 (Moderate)*: Failures isolated to sub-task branches; manual intervention or retry triggers; unconstrained filesystem writes.
|
|
609
|
+
- *5.0 (Optimal)*: Isolated specialist filesystem slots; schema-enforced handoff reconciliation; compile-time acyclic staging; sub-second VM/checkpoint rollback; deny-by-default path boundaries.
|
|
610
|
+
4. **Orchestration Topology Flexibility (OTF)**: Expressive power of coordination patterns (DAGs, cyclic loops, dynamic swarms, human-in-the-loop).
|
|
611
|
+
- *1.0 (Pathological)*: Rigid, hardcoded linear pipelines; cannot branch, parallelize, or stage subtasks.
|
|
612
|
+
- *3.0 (Moderate)*: Linear SOP pipelines or unconstrained dynamic agent spawning (prone to swarm sprawl and infinite loops).
|
|
613
|
+
- *5.0 (Optimal)*: Full support for arbitrary cyclic statecharts, dynamic branching, map-reduce fan-out, and human-in-the-loop pause/resume semantics.
|
|
614
|
+
5. **OS & Execution Boundary Safety (OES)**: Sandboxing, containerization, least-privilege enforcement, and host isolation.
|
|
615
|
+
- *1.0 (Pathological)*: Arbitrary host command execution without isolation, timeouts, or permission enforcement.
|
|
616
|
+
- *3.0 (Moderate)*: Containerized execution (Docker) with volume mounting or interactive user confirmation prompts.
|
|
617
|
+
- *5.0 (Optimal)*: Dedicated hardware-level microVM (Firecracker KVM) with zero access to host infrastructure.
|
|
618
|
+
|
|
619
|
+
---
|
|
620
|
+
|
|
621
|
+
### 4.2 Comprehensive 8-Harness Quantitative Scoring Matrix
|
|
622
|
+
|
|
623
|
+
The table below presents the authoritative scores for all eight audited harnesses across the five evaluation dimensions:
|
|
624
|
+
|
|
625
|
+
| Harness Audited | Primary Operating Paradigm | TCP (1–5) | ETC (1–5) | FIB (1–5) | OTF (1–5) | OES (1–5) | Composite (/25.0) | Normalized (/5.0) |
|
|
626
|
+
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
627
|
+
| **MetaGPT** | SOP Waterfall DAG (Pub/Sub Message Pool) | 3.0 | 1.0 | 2.0 | 3.0 | 1.0 | **10.0** | **2.0** |
|
|
628
|
+
| **CrewAI** | Role-Playing Collectives (Sequential / Hierarchical) | 2.0 | 1.0 | 2.0 | 2.0 | 1.0 | **8.0** | **1.6** |
|
|
629
|
+
| **AutoGen / Swarm** | Conversational Broadcast / Client-Side Swarm | 1.0 | 1.0 | 1.0 | 3.0 | 3.0 | **9.0** | **1.8** |
|
|
630
|
+
| **LangGraph** | Cyclic Statechart DAGs & Channel Reducers | 4.0 | 2.0 | 4.0 | **5.0** | 2.0 | **17.0** | **3.4** |
|
|
631
|
+
| **Devin / Cognition Sol** | Cloud MicroVM Workspace & Tri-Modal Modalities | 3.5 | 3.0 | **4.8** | 4.0 | **5.0** | **20.3** | **4.06** |
|
|
632
|
+
| **Claude Code / SWE-agent** | Terminal ACI & Interactive Bash Loops | 3.2 | 3.0 | 3.0 | 3.0 | 3.5 | **15.7** | **3.14** |
|
|
633
|
+
| **Antigravity `/teamwork`** | Hierarchical Swarm Fleet & File Conventions | 2.0 | 2.5 | 3.0 | 4.2 | 3.8 | **15.5** | **3.10** |
|
|
634
|
+
| **Agy-Squad (Reference)** | **Staged Manifests + Pointer-Over-Wire + RTK** | **5.0** | **5.0** | 4.6 | 4.5 | 4.4 | **23.5** | **4.70** |
|
|
635
|
+
|
|
636
|
+
---
|
|
637
|
+
|
|
638
|
+
### 4.3 Detailed Architectural Justifications across All 8 Harnesses
|
|
639
|
+
|
|
640
|
+
#### 1. MetaGPT (Composite: 10.0/25.0 | Normalized: 2.0/5.0)
|
|
641
|
+
- **TCP (3.0)**: SOP decomposition segments work into discrete roles, avoiding monolithic prompt bloat. However, roles receive entire uncompressed upstream documents (1,500–4,000 tokens), and MetaGPT provides no out-of-band pointer mechanics.
|
|
642
|
+
- **ETC (1.0)**: Writes plain files to workspace directories. Provides zero SHA-256 digests, no Merkle trees, and no cryptographic verification.
|
|
643
|
+
- **FIB (2.0)**: Upstream errors propagate directly into downstream code (GIGO). Action retries handle local JSON/Pydantic parse errors, but published corrupt messages in `Environment.memory` cannot be rolled back.
|
|
644
|
+
- **OTF (3.0)**: Highly effective for linear SOP pipelines (Waterfall), but poorly suited for arbitrary non-linear graphs, dynamic peer swarms, or branching map-reduce workflows.
|
|
645
|
+
- **OES (1.0)**: Executes generated code and tests directly on the host OS via Python `subprocess.run()`. Unsafe execution boundary by default.
|
|
646
|
+
|
|
647
|
+
#### 2. CrewAI (Composite: 8.0/25.0 | Normalized: 1.6/5.0)
|
|
648
|
+
- **TCP (2.0)**: Sequential tasks inject full raw string outputs of preceding tasks into prompt templates. While integrating ChromaDB vector memory, prompt token bloat compounds rapidly in sequential and hierarchical runs.
|
|
649
|
+
- **ETC (1.0)**: Writes unstructured text or Pydantic JSON to disk. Zero cryptographic integrity checks or tamper-evidence.
|
|
650
|
+
- **FIB (2.0)**: In sequential mode, an error in Task $N$ invalidates Task $N+1$ or crashes the run. In hierarchical mode, coworker hallucinations trigger repetitive manager query loops that exhaust token budgets.
|
|
651
|
+
- **OTF (2.0)**: Rigidly constrained to linear sequential arrays (`Process.sequential`) or single-manager hierarchical delegation (`Process.hierarchical`). Lacks cyclic graph routing and concurrent branch-and-merge.
|
|
652
|
+
- **OES (1.0)**: Executes custom tools directly inside the host Python process without containerization or permission gating.
|
|
653
|
+
|
|
654
|
+
#### 3. AutoGen & OpenAI Swarm (Composite: 9.0/25.0 | Normalized: 1.8/5.0)
|
|
655
|
+
- **TCP (1.0)**: Suffers from catastrophic conversational transcript broadcast. In AutoGen `GroupChat`, all $N$ agents receive the entire chat history on every turn ($O(N^2)$ token overhead). Swarm passes flat unpruned message lists across function handoffs. Rapid context exhaustion is guaranteed on multi-turn tasks.
|
|
656
|
+
- **ETC (1.0)**: Outputs are transient conversational strings or unhashed local files. Zero cryptographic attestation.
|
|
657
|
+
- **FIB (1.0)**: Shared conversational broadcast creates an undivided failure domain. A hallucination by one agent is broadcast to all participants, often triggering collective hallucination cascades or polite infinite chatter loops.
|
|
658
|
+
- **OTF (3.0)**: Flexible conversational topologies (broadcast group chat, 1-on-1 ping-pong, dynamic supervisor speaker selection) and Swarm's decentralized function-driven handoffs. However, lacks formal DAG validation and structured graph semantics.
|
|
659
|
+
- **OES (3.0)**: AutoGen includes built-in `DockerCommandLineCodeExecutor`, enabling sandboxed code execution inside isolated Docker containers. (Swarm lacks sandboxing, but AutoGen's native Docker integration establishes a baseline of 3.0).
|
|
660
|
+
|
|
661
|
+
#### 4. LangGraph (Composite: 17.0/25.0 | Normalized: 3.4/5.0)
|
|
662
|
+
- **TCP (4.0)**: Decouples persistent execution state from the LLM prompt context window. Channels allow nodes to exchange concise state deltas. Message trimming (`trim_messages`) and filtering primitives allow fine-grained prompt window control. Deducted 1 point because the popular `messages: Annotated[list, add_messages]` pattern defaults to unpruned accumulation unless manually configured.
|
|
663
|
+
- **ETC (2.0)**: Checkpointers record execution history and DAG state lineages (`thread_id`, `checkpoint_id`, `parent_checkpoint_id`). Provides strong operational traceability through LangSmith, but lacks content-addressable SHA-256 hashing, Merkle trees, or digital signatures over deliverables. State can be mutated in Postgres/SQLite without cryptographic detection.
|
|
664
|
+
- **FIB (4.0)**: Exceptional structural isolation. State transitions occur at node boundaries; checkpointers enable exact rollback and time-travel debugging. Human-in-the-loop interruption barriers (`interrupt_before`, `interrupt_after`) stop errors before they propagate to critical systems. Deducted 1 point because semantic data errors in shared state keys can still propagate across nodes.
|
|
665
|
+
- **OTF (5.0)**: The industry benchmark for orchestration flexibility. Fully supports cyclic graphs, Statecharts, conditional routing, subgraphs, parallel map-reduce via `Send`, dynamic branch-and-merge, and human-in-the-loop pause/resume semantics.
|
|
666
|
+
- **OES (2.0)**: Does not provide native execution sandboxing; executes tools in the host process unless the developer brings external sandbox infrastructure.
|
|
667
|
+
|
|
668
|
+
#### 5. Devin / Cognition Sol (Composite: 20.3/25.0 | Normalized: 4.06/5.0)
|
|
669
|
+
- **TCP (3.5)**: Uses observation summarization and filtered terminal outputs. However, long-horizon cloud sessions still accumulate substantial prompt history across complex multi-hour tasks.
|
|
670
|
+
- **ETC (3.0)**: Tracks git commits, PR artifacts, and terminal logs. Does not compute cryptographic SHA-256 bundles linking exact source inputs to specialist reviews.
|
|
671
|
+
- **FIB (4.8)**: Near-perfect runtime isolation via KVM microVMs. CoW snapshots enable rolling back broken states to exact block checkpoints.
|
|
672
|
+
- **OTF (4.0)**: Sophisticated internal hierarchical planner capable of managing sub-goals, though closed-source and fixed to Cognition's cloud control plane.
|
|
673
|
+
- **OES (5.0)**: Gold standard. Dedicated Firecracker microVM kernel, hardware isolation, and zero access to host infrastructure.
|
|
674
|
+
|
|
675
|
+
#### 6. Claude Code & SWE-agent (Composite: 15.7/25.0 | Normalized: 3.14/5.0)
|
|
676
|
+
- **TCP (3.2)**: ACI windowed viewing (SWE-agent) and targeted diff editing (Claude Code) prevent full-file dumps. However, `/compact` is lossy, stripping older tool contexts and inducing prompt cache invalidation.
|
|
677
|
+
- **ETC (3.0)**: Captures git diffs and test exit codes cleanly. Relies on standard git commits without out-of-band cryptographic digest manifests.
|
|
678
|
+
- **FIB (3.0)**: Single-agent execution loop. If the model makes a bad refactoring choice, it must manually revert changes via git or edit commands; errors can compound across turns.
|
|
679
|
+
- **OTF (3.0)**: Primarily optimized for single-agent iterative problem solving. Lacks native multi-agent cohort compilation or specialist delegation.
|
|
680
|
+
- **OES (3.5)**: Direct host shell execution gated by user confirmation prompts. Optional Docker containerization mitigates host damage but requires manual container configuration.
|
|
681
|
+
|
|
682
|
+
#### 7. Google Antigravity Native `/teamwork` (Composite: 15.5/25.0 | Normalized: 3.10/5.0)
|
|
683
|
+
- **TCP (2.0)**: Severe token bloat. Streams conversational status updates and multi-KB reports directly into parent prompt contexts (4,940 bytes across 5 wakeups in benchmark). Deploys redundant review fleets consuming 228.6 KB of transcript.
|
|
684
|
+
- **ETC (2.5)**: Generates structured markdown reports (`handoff.md`, `progress.md`), but lacks cryptographic hashing. Verification is delegated to expensive, non-deterministic LLM subagents.
|
|
685
|
+
- **FIB (3.0)**: Subagents execute in separate conversations, providing basic process isolation. However, lack of report slot reservations can lead to file overwriting or abandoned files.
|
|
686
|
+
- **OTF (4.2)**: Highly flexible native orchestration supporting wrapper agents, orchestrators, explorers, report writers, reviewers, and auditors.
|
|
687
|
+
- **OES (3.8)**: Antigravity IDE sandbox with execution policies and reactive wakeups. Lacks automated teardown contracts, leaving 41+ untracked files in the workspace.
|
|
688
|
+
|
|
689
|
+
#### 8. Agy-Squad Reference Baseline (Composite: 23.5/25.0 | Normalized: 4.70/5.0)
|
|
690
|
+
- **TCP (5.0)**: Industry-leading. Pointer-Over-Wire restricts IPC messages to $\le 500\text{ B}$ (saving $95.0\%$ of parent wakeup bytes). RTK transparently strips passing test logs (saving $839.3\text{K}$ tokens). 3-tier inspection enforces bounded viewing.
|
|
691
|
+
- **ETC (5.0)**: Deterministic SHA-256 digest bundles (`report.sources.json`) hash all input source files, specialist reports, and lead synthesis. Verified in $20.0\text{ ms}$ at zero token cost.
|
|
692
|
+
- **FIB (4.6)**: Compile-time acyclic staging (`dispatch_squad`) with pre-allocated report slots. Specialists cannot clobber peer reports. Reconciles handoffs against strict expected schemas.
|
|
693
|
+
- **OTF (4.5)**: Staged compiler supports flat cohorts (`all_l1` lead stage) and multi-specialist cohorts (`ab_audit`), with task-proportional scaling (3 agents vs 8 in benchmark).
|
|
694
|
+
- **OES (4.4)**: Sandboxed CLI session (`bin/agy-squad.mjs`) with strict settings permissions (deny-by-default writes to manifests and plugins) and governed `L41` lifecycle teardown (0 untracked Git files). Yields slightly to Devin's KVM hardware virtualization.
|
|
695
|
+
|
|
696
|
+
---
|
|
697
|
+
|
|
698
|
+
## 5. Empirical Evidence & Protocol Tracing (R3)
|
|
699
|
+
|
|
700
|
+
### 5.1 Formal Mathematical Complexity Models ($O(N^2)$ vs $O(N \cdot |S|)$ vs $O(N)$)
|
|
701
|
+
|
|
702
|
+
To substantiate why conversational broadcast architectures experience rapid context exhaustion while Pointer-Over-Wire preserves context indefinitely, we formalize the token and byte consumption models across coordination paradigms:
|
|
703
|
+
|
|
704
|
+
Let:
|
|
705
|
+
- $N$ = Number of participating agents.
|
|
706
|
+
- $M$ = Total coordination steps / turns across the session.
|
|
707
|
+
- $\bar{L}$ = Average message or artifact payload size in tokens ($\bar{L} \approx 500\text{ to }3,000$).
|
|
708
|
+
- $|S|$ = Average size of a structured state dictionary in tokens ($|S| \approx 100\text{ to }500$).
|
|
709
|
+
- $P$ = Pointer receipt size in tokens ($P \le 125\text{ tokens}$, corresponding to $\le 500\text{ bytes}$).
|
|
710
|
+
|
|
711
|
+
#### Complexity Derivations:
|
|
712
|
+
|
|
713
|
+
1. **Full-Transcript Conversational Chatter (AutoGen / Antigravity Native / Swarm)**:
|
|
714
|
+
In conversational broadcast, every agent receives the complete accumulated history on every turn $m$:
|
|
715
|
+
$$T_{\text{cum}}^{\text{broadcast}}(M) = N \sum_{m=1}^{M} (m \cdot \bar{L}) = \frac{N \cdot M(M+1)}{2} \bar{L} = \Theta(N \cdot M^2 \cdot \bar{L}) \implies \mathcal{O}(N \cdot M^2)$$
|
|
716
|
+
Token consumption compounds quadratically with turn count $M$. In a 5-agent group with 40 turns, token spend explodes super-linearly, quickly evicting prompt caches.
|
|
717
|
+
|
|
718
|
+
2. **Structured State Passing (MetaGPT / LangGraph State Dict)**:
|
|
719
|
+
In structured state passing, nodes exchange state updates or full state dictionaries of size $|S|$:
|
|
720
|
+
$$T_{\text{cum}}^{\text{state}}(M) = \sum_{m=1}^{M} |S_m| = \Theta(M \cdot |S|) \implies \mathcal{O}(M \cdot |S|)$$
|
|
721
|
+
Growth is linear with turn count, but scales directly with the cardinality and volume of the shared state dictionary $|S|$.
|
|
722
|
+
|
|
723
|
+
3. **Pointer-Over-Wire Receipts (Agy-Squad Reference)**:
|
|
724
|
+
In Pointer-Over-Wire, inter-agent signaling is strictly bounded to a constant receipt $P \le 500\text{ bytes}$ ($\approx 35\text{--}125\text{ tokens}$), while deliverables are written out-of-band to disk. The lead performs a single bounded verification read ($L_{\text{verify}}$):
|
|
725
|
+
$$T_{\text{cum}}^{\text{pointer}}(M) = \sum_{m=1}^{M} P_m + L_{\text{verify}} \le M \cdot P_{\max} + L_{\text{verify}} = \Theta(M) \implies \mathcal{O}(M)$$
|
|
726
|
+
Communication overhead is strictly linear with a tiny constant factor ($P \ll \bar{L}$).
|
|
727
|
+
|
|
728
|
+
```
|
|
729
|
+
Token Consumption Growth vs. Orchestration Turns
|
|
730
|
+
Cumulative Tokens
|
|
731
|
+
^
|
|
732
|
+
| / (AutoGen / Chatter: O(N * M^2))
|
|
733
|
+
| /
|
|
734
|
+
| /
|
|
735
|
+
| / / (CrewAI Sequential: O(M^2))
|
|
736
|
+
| / /
|
|
737
|
+
| / /
|
|
738
|
+
| / / / (MetaGPT SOP: O(M * L))
|
|
739
|
+
| / / /
|
|
740
|
+
| / / / ------------------ (LangGraph State Dict: O(M * |S|))
|
|
741
|
+
| / / /
|
|
742
|
+
| / / /
|
|
743
|
+
| / / /
|
|
744
|
+
| / / / ........................ (Agy-Squad Pointer: O(M * P), P <= 500B)
|
|
745
|
+
+------------------------------------------------------------>
|
|
746
|
+
Turns (M)
|
|
747
|
+
```
|
|
748
|
+
|
|
749
|
+
---
|
|
750
|
+
|
|
751
|
+
### 5.2 Empirical Protocol Telemetry & Head-to-Head Comparison
|
|
752
|
+
|
|
753
|
+
Our empirical findings are grounded in the head-to-head empirical benchmark conducted on November 8, 2026 ([`benchmarks/teamwork-vs-squad/`](file:///home/paragon-ux/Antigravity/Agy-Squad/benchmarks/teamwork-vs-squad/README.md) and [`docs/benchmarks.md`](file:///home/paragon-ux/Antigravity/Agy-Squad/docs/benchmarks.md)) across identical checkout states of the 6-defect two-domain webhook relay fixture (`gateway/` and `worker/`):
|
|
754
|
+
|
|
755
|
+
| Recorded Telemetry Dimension | Antigravity Native `/teamwork` (Control Arm) | Agy-Squad Reference Arm (Pointer-Over-Wire + RTK) | Delta / Architectural Improvement |
|
|
756
|
+
| :--- | :---: | :---: | :---: |
|
|
757
|
+
| **Planted Defect Recall** | `6 / 6` (100%) | `6 / 6` (100%) | **Parity** (Both caught all 6 defects) |
|
|
758
|
+
| **Subagents Spawned** | 8 agents | 3 agents (1 lead, 2 specialists) | **$62.5\%$ reduction** (3 vs. 8) |
|
|
759
|
+
| **Total Transcript Steps** | 645 steps | 64 steps | **$90.1\%$ reduction** ($10.1\times$ fewer steps) |
|
|
760
|
+
| **Total Assistant LLM Turns** | 321 turns | 26 turns | **$91.9\%$ reduction** ($12.3\times$ fewer turns) |
|
|
761
|
+
| **Total Tool Calls** | 302 calls | 32 calls | **$89.4\%$ reduction** ($9.4\times$ fewer calls) |
|
|
762
|
+
| **Raw Transcript Volume** | 817.4 KB | 148.2 KB | **$81.9\%$ reduction** ($669.2\text{ KB}$ saved) |
|
|
763
|
+
| **Active Wall-Clock Duration** | 1,579s (26m 19s) | 375s (6m 15s) | **$76.2\%$ speedup** (**$4.2\times$ faster**) |
|
|
764
|
+
| **Parent Injected Context Bytes** | 4,940 bytes | 249 bytes | **$95.0\%$ reduction** ($249\text{ B}$ vs. $4,940\text{ B}$) |
|
|
765
|
+
| **Parent Wakeup Events** | 5 wakeups | 1 wakeup | **$80.0\%$ reduction** (1 vs. 5 wakeups) |
|
|
766
|
+
| **Test Suite Output Compaction (RTK)** | ~1,860 tokens/run | ~120 tokens/run | **839.3K tokens saved** ($31.4\%$ net reduction) |
|
|
767
|
+
| **Artifact Verification Latency** | 174 steps (228.6 KB) | 20.0 ms | **Four orders of magnitude faster**; 0 tokens |
|
|
768
|
+
| **Post-Run Git Artifact Drift** | 41 files (207.9 KB) | 0 files (0 KB) | **$100\%$ clean teardown** (`L41` contract) |
|
|
769
|
+
|
|
770
|
+
---
|
|
771
|
+
|
|
772
|
+
### 5.3 Concrete Wire Protocol Trace Analysis: Verbatim Packet Comparison
|
|
773
|
+
|
|
774
|
+
To illustrate how wire payload bloat degrades execution in practice, we examine the exact bytes transmitted during the detection of Seeded Defect `N1` (timing attack and crash DoS in `gateway.mjs:10`):
|
|
775
|
+
|
|
776
|
+
#### Paradigm A: Conversational Broadcast (Antigravity `/teamwork` / AutoGen)
|
|
777
|
+
The worker broadcasts its intermediate discovery directly into the parent conversation:
|
|
778
|
+
```markdown
|
|
779
|
+
[PARENT WAKEUP #3 - 1,842 Bytes Injected into Prompt Context]
|
|
780
|
+
<SYSTEM_MESSAGE>
|
|
781
|
+
[Message] timestamp=2026-11-08T14:22:10Z sender=explorer_gateway_1 priority=MESSAGE_PRIORITY_NORMAL content=
|
|
782
|
+
I have completed my investigation of the gateway module. Here is my complete analysis:
|
|
783
|
+
# Gateway Security & Rate-Limit Audit Report
|
|
784
|
+
## Executive Summary
|
|
785
|
+
I examined fixture/gateway/gateway.mjs and fixture/config.json. The unit tests pass because they only test happy-path valid signatures. However, there is a critical vulnerability in verifyWebhookSignature:
|
|
786
|
+
```javascript
|
|
787
|
+
export function verifyWebhookSignature(payload, signature, secret) {
|
|
788
|
+
const expectedBuf = Buffer.from(signature, 'hex');
|
|
789
|
+
const actualBuf = crypto.createHmac('sha256', secret).update(payload).digest();
|
|
790
|
+
return crypto.timingSafeEqual(actualBuf, expectedBuf);
|
|
791
|
+
}
|
|
792
|
+
```
|
|
793
|
+
If signature is shorter or longer than 64 hex characters (32 bytes), Buffer.from creates a buffer of mismatched length, causing crypto.timingSafeEqual to throw an unhandled RangeError crash DoS. Furthermore, timingSafeEqual leaks timing data if lengths diverge.
|
|
794
|
+
Remediation: Guard actualBuf.length === expectedBuf.length before comparing.
|
|
795
|
+
Please let me know if you want me to inspect other files or proceed to write the final deliverable.
|
|
796
|
+
</SYSTEM_MESSAGE>
|
|
797
|
+
```
|
|
798
|
+
*Wire Impact:* **$1,842\text{ Bytes}$** ($\approx 490\text{ tokens}$). Re-injected into the parent prompt. Across 5 wakeups, this chatter injected **$4,940\text{ Bytes}$**, diluting attention and causing KV cache misses.
|
|
799
|
+
|
|
800
|
+
#### Paradigm B: Structured State Dict Passing (MetaGPT / LangGraph)
|
|
801
|
+
The worker node returns a state delta dictionary over the graph channel:
|
|
802
|
+
```json
|
|
803
|
+
{
|
|
804
|
+
"node": "gateway_auditor",
|
|
805
|
+
"state_delta": {
|
|
806
|
+
"module": "gateway",
|
|
807
|
+
"defects_found": [
|
|
808
|
+
{
|
|
809
|
+
"id": "SEC-01",
|
|
810
|
+
"severity": "P1",
|
|
811
|
+
"file": "fixture/gateway/gateway.mjs",
|
|
812
|
+
"line": 10,
|
|
813
|
+
"symbol": "verifyWebhookSignature",
|
|
814
|
+
"root_cause": "crypto.timingSafeEqual throws RangeError on buffer length mismatch",
|
|
815
|
+
"code_snippet": "export function verifyWebhookSignature(payload, signature, secret) {\n const expectedBuf = Buffer.from(signature, 'hex');\n const actualBuf = crypto.createHmac('sha256', secret).update(payload).digest();\n return crypto.timingSafeEqual(actualBuf, expectedBuf);\n}",
|
|
816
|
+
"remediation": "Add actualBuf.length === expectedBuf.length guard before timingSafeEqual"
|
|
817
|
+
}
|
|
818
|
+
],
|
|
819
|
+
"full_report_markdown": "# Gateway Security Audit\n\n[... Complete 4.5 KB report text ...]"
|
|
820
|
+
}
|
|
821
|
+
}
|
|
822
|
+
```
|
|
823
|
+
*Wire Impact:* **$4,820\text{ Bytes}$** ($\approx 1,220\text{ tokens}$). Transmitted over the graph edge to all downstream subscriber nodes.
|
|
824
|
+
|
|
825
|
+
#### Paradigm C: Agy-Squad Pointer-Over-Wire Receipt
|
|
826
|
+
The specialist writes its complete markdown analysis directly to its pre-allocated slot (`scratch/.../specialist-1/report.md`) and emits only a single-line receipt:
|
|
827
|
+
```text
|
|
828
|
+
READY scratch/agy-squad/run-f1465008/plan-07bfb98d/squad-ab_audit/specialist-1/report.md — verifyWebhookSignature throws RangeError on length mismatch in gateway.mjs:10 (P1 crash DoS)
|
|
829
|
+
```
|
|
830
|
+
*Wire Impact:* **$208\text{ Bytes}$** ($\approx 52\text{ tokens}$).
|
|
831
|
+
- Wire payload is strictly single-line and well within the $500\text{ B}$ ceiling (`mcp/handoffs.mjs:103`).
|
|
832
|
+
- The parent receives exactly 1 wakeup receipt ($249\text{ B}$) upon squad completion.
|
|
833
|
+
- Net communication wire savings vs. Paradigm A: **$95.0\%$ byte reduction**.
|
|
834
|
+
|
|
835
|
+
---
|
|
836
|
+
|
|
837
|
+
### 5.4 Pointer-Over-Wire Grammar & Invariant Contracts
|
|
838
|
+
|
|
839
|
+
In [`mcp/handoffs.mjs:103-125`](file:///home/paragon-ux/Antigravity/Agy-Squad/mcp/handoffs.mjs), Agy-Squad enforces the formal grammar and length invariants of the Pointer-Over-Wire protocol:
|
|
840
|
+
|
|
841
|
+
```javascript
|
|
842
|
+
// mcp/handoffs.mjs:6, 103-125
|
|
843
|
+
const messageLimit = 500;
|
|
844
|
+
|
|
845
|
+
if (typeof message !== "string" || Buffer.byteLength(message) > messageLimit || /[\r\n\u2028\u2029]/.test(message)) {
|
|
846
|
+
throw new Error("Handoff must be one line within 500 UTF-8 bytes.");
|
|
847
|
+
}
|
|
848
|
+
const readyPrefix = `READY ${report} — `;
|
|
849
|
+
const partialPrefix = `PARTIAL ${report} — `;
|
|
850
|
+
const status = message.startsWith(readyPrefix)
|
|
851
|
+
? "READY"
|
|
852
|
+
: message.startsWith(partialPrefix)
|
|
853
|
+
? "PARTIAL"
|
|
854
|
+
: undefined;
|
|
855
|
+
const observation = status
|
|
856
|
+
? message.slice((status === "READY" ? readyPrefix : partialPrefix).length)
|
|
857
|
+
: "";
|
|
858
|
+
const noReport = /^NO_REPORT — \S/.test(message);
|
|
859
|
+
if (!noReport && (!status || !observation.trim())) {
|
|
860
|
+
throw new Error("Handoff must include the sender's assigned report and an observation, or a NO_REPORT reason.");
|
|
861
|
+
}
|
|
862
|
+
```
|
|
863
|
+
|
|
864
|
+
#### Protocol Grammar Rules:
|
|
865
|
+
1. `READY <assigned-report-slot> — <observation>`: Specialist has verified code/tests and published complete markdown evidence to its assigned slot.
|
|
866
|
+
2. `PARTIAL <assigned-report-slot> — <blocker-reason>`: Specialist completed partial work but encountered an environmental barrier; partial evidence recorded.
|
|
867
|
+
3. `NO_REPORT — <fatal-reason>`: Specialist was unable to produce evidence (e.g. invalid arguments or missing prerequisites).
|
|
868
|
+
4. **Length & Structure Invariants**:
|
|
869
|
+
- `Buffer.byteLength(message) <= 500` (Strict UTF-8 byte boundary).
|
|
870
|
+
- Zero newlines: `!/[\r\n\u2028\u2029]/.test(message)`. Multiline messages are rejected.
|
|
871
|
+
- Slot Matching: The path in the receipt must match the exact pre-allocated slot registered for that specialist.
|
|
872
|
+
|
|
873
|
+
---
|
|
874
|
+
|
|
875
|
+
### 5.5 Cryptographic Bundle Manifest Specification (`report.sources.json`)
|
|
876
|
+
|
|
877
|
+
Upon receiving verified `READY` receipts from all specialists in a cohort, the Squad Lead executes `mcp/handoffs.mjs bundle`, which constructs the immutable cryptographic bundle:
|
|
878
|
+
|
|
879
|
+
```json
|
|
880
|
+
{
|
|
881
|
+
"squadId": "ab_audit",
|
|
882
|
+
"verdict": "PASS",
|
|
883
|
+
"synthesis": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
|
884
|
+
"verifiedFiles": {
|
|
885
|
+
"fixture/gateway/gateway.mjs": "a1b2c3d4e5f60718293a4b5c6d7e8f90123456789abcdef0123456789abcdef0",
|
|
886
|
+
"fixture/worker/worker.mjs": "d4e5f6a1b2c30718293a4b5c6d7e8f90123456789abcdef0123456789abcdef0",
|
|
887
|
+
"fixture/config.json": "789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567"
|
|
888
|
+
},
|
|
889
|
+
"reports": [
|
|
890
|
+
{
|
|
891
|
+
"path": "scratch/agy-squad/run-f1465008/plan-07bfb98d/squad-ab_audit/specialist-1/report.md",
|
|
892
|
+
"sha256": "4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a",
|
|
893
|
+
"markdown": "# Gateway Security & Contract Audit Report\n..."
|
|
894
|
+
},
|
|
895
|
+
{
|
|
896
|
+
"path": "scratch/agy-squad/run-f1465008/plan-07bfb98d/squad-ab_audit/specialist-2/report.md",
|
|
897
|
+
"sha256": "ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d",
|
|
898
|
+
"markdown": "# Worker Reliability & State Audit Report\n..."
|
|
899
|
+
}
|
|
900
|
+
]
|
|
901
|
+
}
|
|
902
|
+
```
|
|
903
|
+
|
|
904
|
+
#### Bundling Guarantees:
|
|
905
|
+
- **Source Pinning (`verifiedFiles`)**: Every primary source file evaluated by the cohort is hashed via SHA-256. If a source file is edited after verification, `mcp/handoffs.mjs verify` detects the hash mismatch and fails.
|
|
906
|
+
- **Synthesis Digest (`synthesis`)**: Computes the SHA-256 hash of the lead's final report (`lead/report.md`).
|
|
907
|
+
- **Unambiguous PASS Enforcement**: `readSynthesis` strips markdown code blocks and backticks, evaluates natural language negations ("not PASS", "didn't PASS"), and requires an unambiguous `PASS`.
|
|
908
|
+
- **Atomic Publication**: Written via exclusive temporary files (`0o600`, `O_EXCL`), flushed to disk via `fs.fsyncSync`, and atomically linked into `lead/report.sources.json`.
|
|
909
|
+
- **Zero-Token Verification**: Independent agents or CI/CD pipelines verify the bundle in **$20.0\text{ ms}$** without querying an LLM.
|
|
910
|
+
|
|
911
|
+
---
|
|
912
|
+
|
|
913
|
+
## 6. Synthesized Architectural Matrix & Pareto Frontier (R4)
|
|
914
|
+
|
|
915
|
+
### 6.1 Exhaustive Multi-Dimensional Comparative Trade-Off Matrix
|
|
916
|
+
|
|
917
|
+
The matrix below (Table 6.1: Synthesized Comparative Architecture Matrix) synthesizes the trade-offs, strengths, and operational characteristics of all eight audited multi-agent harnesses:
|
|
918
|
+
|
|
919
|
+
| Architectural Dimension | MetaGPT | CrewAI | AutoGen / Swarm | LangGraph | Devin / Sol | Claude Code | Antigravity `/teamwork` | Agy-Squad (Reference) |
|
|
920
|
+
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|
|
921
|
+
| **Primary Coordination Model** | SOP Waterfall DAG (Pub/Sub Pool) | Sequential / Hierarchical Crew | Conversational Broadcast / Swarm | Cyclic Statechart Graph (DAG / Loops) | Cloud MicroVM Control Plane | Terminal ACI & Interactive REPL | Hierarchical Swarm Fleet | **Staged Lead + Scoped Specialists** |
|
|
922
|
+
| **Signaling Wire Protocol** | Schema Pydantic in Memory | Raw String Concatenation | Full Transcript Message Array | State Channel Updates | Filtered Observation RPC | JSON-RPC Tool Requests | Conversational Wakeups (`<SYSTEM_MESSAGE>`) | **Pointer-Over-Wire Receipts ($\le 500\text{ B}$)** |
|
|
923
|
+
| **Wire Overhead per Turn** | $1.5\text{K}–4\text{K}$ tokens | Unbounded raw text | $O(N^2)$ quadratic bloat | $O(M \cdot \vert S \vert)$ state dicts | Filtered stdout snippets | Bounded diffs & ACI | $4,940\text{ B}$ across 5 wakeups | **$249\text{ B}$ single-line receipt** |
|
|
924
|
+
| **Token Preservation** | Medium (SOP stages) | Low (prompt string dumps) | Very Low (quadratic chat) | High (state decoupled from prompt) | Medium-High (observation filter) | High (ACI windowing, diffs) | Low (conversational wakeups) | **Optimal (Pointer wire + RTK)** |
|
|
925
|
+
| **Evidence Traceability** | None (plain files) | None (plain files) | None (transient chat) | Medium (checkpoint IDs in DB) | Medium (git commits, PRs) | Medium (git patches) | Medium (markdown reports) | **Cryptographic (SHA-256 manifest bundles)** |
|
|
926
|
+
| **Verification Mechanism** | Semantic QA prompts | Semantic Review tools | Prompt critic | Conditional edge gates | Semantic self-eval & tests | Test execution & user | Review fleet (174 steps, 228 KB) | **Algorithmic $20\text{ ms}$ digest check (0 tok)** |
|
|
927
|
+
| **Failure Isolation** | Low-Medium (GIGO cascade) | Low (manager query loops) | Very Low (chat cascades) | High (checkpoint rollback, HITL) | **Optimal (KVM microVM CoW snapshots)** | Medium (single-agent manual undo) | Medium (separate subagent sessions) | **High (isolated scratch slots, schema check)** |
|
|
928
|
+
| **Topology Flexibility** | Medium (SOP waterfall) | Low (sequential / manager) | Medium (chat broadcast / swarm) | **Maximum (arbitrary cyclic statecharts)** | High (hierarchical planning) | Low (single-agent interactive loop) | High (hierarchical fleet) | **High (staged manifest cohorts)** |
|
|
929
|
+
| **Execution Sandboxing** | None (host `subprocess`) | None (host Python) | Native Docker (AutoGen) | None (host Python) | **Hardware KVM MicroVM (Firecracker)** | Host shell / Optional Docker | Host IDE sandbox | **WSL2 Substrate + Deny-by-default Sandbox** |
|
|
930
|
+
| **Teardown Cleanliness** | Unmanaged files | Unmanaged files | Unmanaged containers/files | Persistent DB records | Ephemeral VM destroyed | Working tree changes | 41 untracked files ($207.9\text{ KB}$) | **Governed `L41` Teardown ($0$ untracked files)** |
|
|
931
|
+
| **Zero-Config Onboarding & Ad-Hoc Ergonomics** | Medium (requires SOP configs & role schemas) | Medium (requires Python agent/task scripts) | Medium-Low (scripted Python chat setup) | Low (requires graph construction & compilation) | Medium (cloud provisioning & repo cloning) | **Optimal (Immediate zero-config CLI REPL in ambient shell)** | Medium (IDE extension & prompt setup) | **Yields to Claude Code & SWE-agent (staged manifest, scratch slot, & WSL2/PTY daemon overhead)** |
|
|
932
|
+
|
|
933
|
+
---
|
|
934
|
+
|
|
935
|
+
### 6.2 Pareto Frontier Analysis: Where Agy-Squad Dominates
|
|
936
|
+
|
|
937
|
+
Agy-Squad occupies the **Pareto frontier** across three critical systems-engineering dimensions:
|
|
938
|
+
|
|
939
|
+
```
|
|
940
|
+
THE PARETO FRONTIER
|
|
941
|
+
Token & Context Preservation
|
|
942
|
+
|
|
|
943
|
+
| * Agy-Squad (5.0, 5.0)
|
|
944
|
+
|
|
|
945
|
+
|
|
|
946
|
+
| * LangGraph (4.0, 2.0)
|
|
947
|
+
|
|
|
948
|
+
| * Claude Code (3.2, 3.0)
|
|
949
|
+
| * Devin (3.5, 3.0)
|
|
950
|
+
|
|
|
951
|
+
| * MetaGPT (3.0, 1.0)
|
|
952
|
+
| * Antigravity /teamwork (2.0, 2.5)
|
|
953
|
+
| * CrewAI (2.0, 1.0)
|
|
954
|
+
| * AutoGen (1.0, 1.0)
|
|
955
|
+
+-------------------------------- Verifiability
|
|
956
|
+
& Crypto Proof
|
|
957
|
+
```
|
|
958
|
+
|
|
959
|
+
1. **Token Preservation & Communication Wire Overhead**:
|
|
960
|
+
- By enforcing the $\le 500\text{ B}$ receipt invariant and decoupling evidentiary storage to disk, Agy-Squad eliminates quadratic transcript inflation.
|
|
961
|
+
- Demonstrates a **$95.0\%$ reduction in parent injected bytes** ($249\text{ B}$ vs. $4,940\text{ B}$) and **$81.9\%$ reduction in raw transcript volume** ($148.2\text{ KB}$ vs. $817.4\text{ KB}$).
|
|
962
|
+
2. **Cryptographic Integrity & Zero-Token Verification**:
|
|
963
|
+
- While all other audited harnesses rely on non-deterministic, expensive LLM-on-LLM reviews (e.g. `/teamwork` spending 174 steps and 228.6 KB on review fleets), Agy-Squad validates deliverable integrity via SHA-256 cryptographic bundling in **$20.0\text{ ms}$ at zero token cost**.
|
|
964
|
+
3. **Deterministic Lifecycle Governance (`L41`)**:
|
|
965
|
+
- Solves repository pollution by automatically pruning intermediate scratch files upon bundle verification, guaranteeing **`0` untracked Git files** left in the repository.
|
|
966
|
+
|
|
967
|
+
---
|
|
968
|
+
|
|
969
|
+
### 6.3 Strategic Yield Points: Where Alternative Harnesses Excel
|
|
970
|
+
|
|
971
|
+
A rigorous architectural audit must identify where an architecture intentionally **yields** to alternative paradigms:
|
|
972
|
+
|
|
973
|
+
### 6.3.1. Yield Point 1: Hardware MicroVM Sandboxing & Block-Level Snapshots (Devin / Cognition Sol)
|
|
974
|
+
- *Where Devin Excels*: Devin executes inside dedicated AWS Firecracker Linux KVM microVMs with Copy-on-Write (CoW) block storage snapshots. It can instantly revert entire virtual machine states—including OS-level package installations, kernel configuration changes, and active system daemons.
|
|
975
|
+
- *Agy-Squad Yield Point*: Agy-Squad operates on the local host operating system (via Linux / WSL2). While it enforces strict path confinement, deny-by-default write settings, and process isolation, it shares the host OS kernel and cannot perform full-system microVM block rollbacks. For executing untrusted, potentially hostile arbitrary code, Devin's hardware virtualization is architecturally superior.
|
|
976
|
+
|
|
977
|
+
### 6.3.2. Yield Point 2: Arbitrary Cyclic Statecharts & Human-in-the-Loop Graphs (LangGraph)
|
|
978
|
+
- *Where LangGraph Excels*: LangGraph supports arbitrary cyclic graph topologies, dynamic conditional loops, subgraphs, parallel map-reduce via `Send`, and programmatic pause/resume checkpoints (`BaseCheckpointSaver`) with interactive human-in-the-loop state mutation.
|
|
979
|
+
- *Agy-Squad Yield Point*: Agy-Squad intentionally enforces **acyclic staged compilation** (`all_l1` lead stage followed by specialist cohorts). While this prevents runaway token spending and non-terminating loops, complex multi-turn workflows requiring dynamic feedback loops or iterative human intervention must be modeled as sequential staged cohorts rather than dynamic cyclic statecharts.
|
|
980
|
+
|
|
981
|
+
### 6.3.3. Yield Point 3: Zero-Configuration Onboarding & Ad-Hoc Ergonomics
|
|
982
|
+
- *Where Claude Code & SWE-agent Excel*: Interactive software engineering harnesses—specifically Anthropic's **Claude Code** and Princeton's **SWE-agent**—outperform Agy-Squad in zero-overhead, ad-hoc terminal adoption and developer ergonomics. A developer can immediately initialize Claude Code or SWE-agent directly inside any ambient repository with zero prerequisite configuration, instant CLI evaluation, and interactive terminal loops. Key architectural strengths include:
|
|
983
|
+
1. **Zero-Configuration Onboarding**: Requires no declarative task manifests, pre-allocated scratch directories, or environment daemon initialization before executing interactive commands.
|
|
984
|
+
2. **Ad-Hoc Developer Ergonomics**: Developers can directly explore codebases, inspect test failures, and execute single-file hotfixes through a native interactive REPL with bounded diff inspection.
|
|
985
|
+
3. **Direct Ambient Shell Access**: Commands execute directly in the developer's pre-existing shell without requiring custom socket multiplexers or WSL2/PTY bridge configurations.
|
|
986
|
+
- *Agy-Squad Yield Point*: Agy-Squad intentionally yields to Claude Code and SWE-agent in zero-overhead, ad-hoc terminal adoption. Agy-Squad's architecture is engineered for high-assurance, multi-agent production orchestration, requiring:
|
|
987
|
+
1. **Upfront Staged Manifest Compilation**: Tasks must be formally specified in declarative JSON schemas (`manifest.json`) and compiled into phased cohorts (`dispatch_squad`) before execution begins.
|
|
988
|
+
2. **Physical Scratch Slot Allocation**: Every agent role requires explicit scratch slot reservation (`.agents/teamwork/<slot>/`) and ledger lifecycle hooks (`L1`–`L41`).
|
|
989
|
+
3. **WSL2 / PTY Daemon Infrastructure**: Confinement and reactive wakeups depend on a Linux/WSL2 substrate and persistent pseudo-terminal daemon bridges (`bin/terminal.py`).
|
|
990
|
+
|
|
991
|
+
**Architectural Trade-Off**: For rapid, ad-hoc terminal queries, quick interactive experimentation, or single-turn bug fixes, Agy-Squad's formal staged compilation introduces noticeable setup friction compared to Claude Code's immediate zero-configuration CLI evaluation. Conversely, for complex multi-agent engineering workflows requiring cryptographic auditability, rigid failure isolation, and token preservation, Agy-Squad's structured overhead provides guarantees that ad-hoc interactive harnesses cannot deliver.
|
|
992
|
+
|
|
993
|
+
### 6.3.4. Yield Point 4: Deep SOP Schema Generation for Greenfield Projects (MetaGPT)
|
|
994
|
+
- *Where MetaGPT Excels*: MetaGPT's deep Pydantic schemas (PRD, System Design, Data API, Sequence Diagram) enforce rigorous waterfall requirements when generating an entire software project from a single prompt.
|
|
995
|
+
- *Agy-Squad Yield Point*: Agy-Squad is an execution and audit harness; it relies on declarative task manifests (`manifest.json`) or parent agent decomposition rather than embedding a built-in greenfield software development waterfall.
|
|
996
|
+
|
|
997
|
+
### 6.3.5. Yield Point 5: Native Docker Isolation (AutoGen)
|
|
998
|
+
- *Where AutoGen Excels*: AutoGen includes built-in `DockerCommandLineCodeExecutor` support out-of-the-box, allowing containerized execution without manual host configuration.
|
|
999
|
+
- *Agy-Squad Yield Point*: Agy-Squad uses native POSIX/WSL2 processes with Antigravity sandbox policies rather than spawning Docker containers per agent (a decision documented in `LEDGER.md#L8–L12` to eliminate container startup latency).
|
|
1000
|
+
|
|
1001
|
+
---
|
|
1002
|
+
|
|
1003
|
+
### 6.4 Conclusion & Architectural Recommendations
|
|
1004
|
+
|
|
1005
|
+
The comparative audit demonstrates that the next generation of multi-agent software engineering systems cannot rely on conversational chat rooms or unpruned message streams. Conflating inter-process signaling with evidentiary data storage leads directly to context window saturation, attention degradation, and prompt cache invalidation.
|
|
1006
|
+
|
|
1007
|
+
**Key Architectural Recommendations for Engineering Teams:**
|
|
1008
|
+
1. **Adopt Pointer-Over-Wire**: Restrict inter-agent messaging to lightweight locator receipts ($\le 500\text{ B}$) and offload heavy reports, code diffs, and execution logs to out-of-band filesystem slots.
|
|
1009
|
+
2. **Decouple Semantic Review from Cryptographic Verification**: Use LLMs to evaluate code logic and test coverage, but verify deliverable completeness, input source alignment, and tamper resistance using deterministic SHA-256 cryptographic bundling.
|
|
1010
|
+
3. **Govern the Execution Substrate**: Implement transparent command interception (RTK) to strip passing test boilerplate and enforce precision file slicing, saving up to $31\%$ of total token spend.
|
|
1011
|
+
4. **Enforce Deterministic Teardown**: Require multi-agent harnesses to clean up temporary agent profiles, manifests, and intermediate scratch files upon task completion to preserve clean repository state.
|
|
1012
|
+
|
|
1013
|
+
---
|
|
1014
|
+
|
|
1015
|
+
## 7. Appendices
|
|
1016
|
+
|
|
1017
|
+
### Appendix A: Multi-Agent Protocol Tracing Harness (MAPT-Harness) Architecture
|
|
1018
|
+
|
|
1019
|
+
To collect quantitative telemetry without manual intervention, the Multi-Agent Protocol Tracing Harness (MAPT-Harness) deploys five coordinated monitoring probes:
|
|
1020
|
+
|
|
1021
|
+
```
|
|
1022
|
+
┌─────────────────────────────────────────────────────────────────────────┐
|
|
1023
|
+
│ MAPT-Harness Protocol Tracing Architecture │
|
|
1024
|
+
│ │
|
|
1025
|
+
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
|
1026
|
+
│ │ Synthetic Workload Driver │ │
|
|
1027
|
+
│ │ Suite 1: 6-Defect Event Relay Suite 2: Multi-File Refactor │ │
|
|
1028
|
+
│ └─────────────────────────────────┬─────────────────────────────────┘ │
|
|
1029
|
+
│ │ │
|
|
1030
|
+
│ ┌──────────────────────────┴──────────────────────────┐ │
|
|
1031
|
+
│ ▼ ▼ │
|
|
1032
|
+
│ ┌──────────────────────────────┐ ┌────────────────────────┐ │
|
|
1033
|
+
│ │ Arm A: Conversational Fleet │ │ Arm B: Pointer-Over- │ │
|
|
1034
|
+
│ │ (AutoGen / Teamwork Control) │ │ Wire (Agy-Squad) │ │
|
|
1035
|
+
│ └──────────────┬───────────────┘ └───────────┬────────────┘ │
|
|
1036
|
+
│ │ │ │
|
|
1037
|
+
│ ▼ ▼ │
|
|
1038
|
+
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
|
1039
|
+
│ │ Protocol Interception & Telemetry Logger Engine │ │
|
|
1040
|
+
│ │ - JSON-RPC Wire Interceptor: Buffer.byteLength(payload) │ │
|
|
1041
|
+
│ │ - tiktoken Token Counter: cl100k_base / o200k_base tokenization │ │
|
|
1042
|
+
│ │ - Context Health Probe: Transcript scraping via Agy MCP tools │ │
|
|
1043
|
+
│ │ - Filesystem Delta Monitor: git status --porcelain tracking │ │
|
|
1044
|
+
│ │ - Cryptographic Verifier: mcp/handoffs.mjs verify microsecond clock│ │
|
|
1045
|
+
│ └─────────────────────────────────┬─────────────────────────────────┘ │
|
|
1046
|
+
│ │ │
|
|
1047
|
+
│ ▼ │
|
|
1048
|
+
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
|
1049
|
+
│ │ Standardized Telemetry Ledger (results/telemetry.json) │ │
|
|
1050
|
+
│ │ - Per-agent step counts, assistant turns, tool calls, raw bytes │ │
|
|
1051
|
+
│ │ - Cumulative token consumption curves across N turns │ │
|
|
1052
|
+
│ │ - Defect recall score (Planted Needle Adjudication Rubric) │ │
|
|
1053
|
+
│ │ - Post-run workspace hygiene & teardown audit (L41 check) │ │
|
|
1054
|
+
│ └───────────────────────────────────────────────────────────────────┘ │
|
|
1055
|
+
└─────────────────────────────────────────────────────────────────────────┘
|
|
1056
|
+
```
|
|
1057
|
+
|
|
1058
|
+
---
|
|
1059
|
+
|
|
1060
|
+
### Appendix B: Seeded Defect Specifications (Event-Relay Benchmark)
|
|
1061
|
+
|
|
1062
|
+
The canonical head-to-head empirical audit evaluates harness recall across six planted defects in `benchmarks/teamwork-vs-squad/fixture/`:
|
|
1063
|
+
|
|
1064
|
+
1. **Needle 1 (`N1` - P1 Critical Security / DoS)**: `gateway.mjs:10` — `verifyWebhookSignature` calls `crypto.timingSafeEqual` without validating buffer length equality, throwing an unhandled `RangeError` crash DoS if signature length diverges.
|
|
1065
|
+
2. **Needle 2 (`N2` - P2 Reliability / Starvation)**: `gateway.mjs:25` — `checkRateLimit` updates the client timestamp before checking the request count, causing timestamp reset starvation and bypassing window expiration.
|
|
1066
|
+
3. **Needle 3 (`N3` - P2 Reliability / Data Loss)**: `gateway.mjs:40` — `processWebhookRequest` fails to validate the `X-Request-Id` header, permitting duplicate webhook processing and breaking idempotency contracts.
|
|
1067
|
+
4. **Needle 4 (`N4` - P1 Cross-Domain Protocol Mismatch)**: `gateway.mjs:43` vs. `worker.mjs:8` — Gateway emits events conforming to schema `"2.0"`, while Worker strictly rejects events not matching schema `"1.0"`.
|
|
1068
|
+
5. **Needle 5 (`N5` - P1 Protocol Unit Mismatch)**: `worker.mjs:12` vs. `config.json:4` — Worker interprets `maxClockSkew` in seconds while `config.json` specifies milliseconds ($300\text{s}$ vs $300\text{ms}$), permitting 5-minute replay attacks.
|
|
1069
|
+
6. **Needle 6 (`N6` - P1 Data Integrity / Off-by-One)**: `ledger.mjs:9-21` & `worker.mjs:16-28` — Ledger overwrite bug truncates historical transaction logs, combined with an off-by-one retry loop dropping failed events silently.
|
|
1070
|
+
|
|
1071
|
+
---
|
|
1072
|
+
|
|
1073
|
+
### Appendix C: Telemetry Schema Specification (`telemetry.schema.json`)
|
|
1074
|
+
|
|
1075
|
+
All empirical benchmark runs emit a standardized JSON ledger validated against the following JSON Schema:
|
|
1076
|
+
|
|
1077
|
+
```json
|
|
1078
|
+
{
|
|
1079
|
+
"$schema": "http://json-schema.org/draft-07/schema#",
|
|
1080
|
+
"title": "AgySquadBenchmarkTelemetry",
|
|
1081
|
+
"type": "object",
|
|
1082
|
+
"required": [
|
|
1083
|
+
"studyType",
|
|
1084
|
+
"timestamp",
|
|
1085
|
+
"environment",
|
|
1086
|
+
"workloadSuite",
|
|
1087
|
+
"arms",
|
|
1088
|
+
"rubricEvaluation",
|
|
1089
|
+
"reproducibility"
|
|
1090
|
+
],
|
|
1091
|
+
"properties": {
|
|
1092
|
+
"studyType": { "type": "string" },
|
|
1093
|
+
"timestamp": { "type": "string", "format": "date-time" },
|
|
1094
|
+
"environment": {
|
|
1095
|
+
"type": "object",
|
|
1096
|
+
"required": ["os", "nodeVersion", "rtkVersion", "agyContextSaverVersion"],
|
|
1097
|
+
"properties": {
|
|
1098
|
+
"os": { "type": "string" },
|
|
1099
|
+
"nodeVersion": { "type": "string" },
|
|
1100
|
+
"rtkVersion": { "type": "string" },
|
|
1101
|
+
"agyContextSaverVersion": { "type": "string" }
|
|
1102
|
+
}
|
|
1103
|
+
},
|
|
1104
|
+
"workloadSuite": { "type": "string" },
|
|
1105
|
+
"arms": {
|
|
1106
|
+
"type": "object",
|
|
1107
|
+
"additionalProperties": {
|
|
1108
|
+
"type": "object",
|
|
1109
|
+
"required": [
|
|
1110
|
+
"subagentsSpawned",
|
|
1111
|
+
"activeWallClockSeconds",
|
|
1112
|
+
"totalSteps",
|
|
1113
|
+
"totalAssistantTurns",
|
|
1114
|
+
"totalToolCalls",
|
|
1115
|
+
"aggregateWireBytes",
|
|
1116
|
+
"cumulativePromptTokens",
|
|
1117
|
+
"parentWakeupBytes",
|
|
1118
|
+
"parentWakeupCount",
|
|
1119
|
+
"untrackedFilesPostRun",
|
|
1120
|
+
"agents"
|
|
1121
|
+
],
|
|
1122
|
+
"properties": {
|
|
1123
|
+
"subagentsSpawned": { "type": "integer" },
|
|
1124
|
+
"activeWallClockSeconds": { "type": "number" },
|
|
1125
|
+
"totalSteps": { "type": "integer" },
|
|
1126
|
+
"totalAssistantTurns": { "type": "integer" },
|
|
1127
|
+
"totalToolCalls": { "type": "integer" },
|
|
1128
|
+
"aggregateWireBytes": { "type": "integer" },
|
|
1129
|
+
"cumulativePromptTokens": { "type": "integer" },
|
|
1130
|
+
"parentWakeupBytes": { "type": "integer" },
|
|
1131
|
+
"parentWakeupCount": { "type": "integer" },
|
|
1132
|
+
"untrackedFilesPostRun": { "type": "integer" },
|
|
1133
|
+
"agents": {
|
|
1134
|
+
"type": "array",
|
|
1135
|
+
"items": {
|
|
1136
|
+
"type": "object",
|
|
1137
|
+
"required": ["role", "conversationId", "steps", "turns", "toolCalls", "transcriptKb"],
|
|
1138
|
+
"properties": {
|
|
1139
|
+
"role": { "type": "string" },
|
|
1140
|
+
"conversationId": { "type": "string" },
|
|
1141
|
+
"steps": { "type": "integer" },
|
|
1142
|
+
"turns": { "type": "integer" },
|
|
1143
|
+
"toolCalls": { "type": "integer" },
|
|
1144
|
+
"transcriptKb": { "type": "number" }
|
|
1145
|
+
}
|
|
1146
|
+
}
|
|
1147
|
+
}
|
|
1148
|
+
}
|
|
1149
|
+
}
|
|
1150
|
+
},
|
|
1151
|
+
"rubricEvaluation": {
|
|
1152
|
+
"type": "object",
|
|
1153
|
+
"required": ["plantedNeedles", "recalledNeedles", "precision", "recall"],
|
|
1154
|
+
"properties": {
|
|
1155
|
+
"plantedNeedles": { "type": "integer" },
|
|
1156
|
+
"recalledNeedles": { "type": "integer" },
|
|
1157
|
+
"precision": { "type": "number" },
|
|
1158
|
+
"recall": { "type": "number" }
|
|
1159
|
+
}
|
|
1160
|
+
},
|
|
1161
|
+
"reproducibility": {
|
|
1162
|
+
"type": "object",
|
|
1163
|
+
"required": ["offlineVerificationCommand", "sha256BundleDigest"],
|
|
1164
|
+
"properties": {
|
|
1165
|
+
"offlineVerificationCommand": { "type": "string" },
|
|
1166
|
+
"sha256BundleDigest": { "type": "string" }
|
|
1167
|
+
}
|
|
1168
|
+
}
|
|
1169
|
+
}
|
|
1170
|
+
}
|
|
1171
|
+
```
|
|
1172
|
+
|
|
1173
|
+
---
|
|
1174
|
+
*End of Architectural Dossier `docs/plans/comparative-harness-architecture-teamwork.md`.*
|