synapse-hub 2.5.0 → 2.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/agents/failure-analyst.md +17 -0
- package/.agents/agents/orchestrator.md +20 -2
- package/.agents/agents/pom-crafter.md +20 -0
- package/.agents/agents/quality-gatekeeper.md +20 -2
- package/.agents/agents/test-runner.md +16 -0
- package/.agents/rules/context-engineering.md +22 -102
- package/.agents/rules/git-push-protocol.md +1 -1
- package/.agents/rules/graphify.md +2 -0
- package/.agents/rules/mandatory-playwright-live.md +9 -9
- package/.agents/rules/mobile_appium_rules.md +27 -116
- package/.agents/rules/performance_testing_rules.md +18 -62
- package/.agents/rules/skill-precedence.md +22 -42
- package/.agents/rules/systematic-debugging.md +10 -29
- package/.agents/skills/actor-qa-swarm/SKILL.md +77 -0
- package/.agents/skills/actor-qa-swarm/examples/broker_sharding_sample.json +12 -0
- package/.agents/skills/agentless-qa-repair/SKILL.md +69 -0
- package/.agents/skills/agentless-qa-repair/examples/healing_report_sample.json +17 -0
- package/.agents/skills/api-spec-to-e2e-matrix/SKILL.md +1 -1
- package/.agents/skills/clean-pom-crafter/SKILL.md +2 -2
- package/.agents/skills/db-state-verifier-and-seeder/SKILL.md +4 -4
- package/.agents/skills/event-stream-qa-orchestrator/SKILL.md +79 -0
- package/.agents/skills/event-stream-qa-orchestrator/examples/timeline_sample.jsonl +2 -0
- package/.agents/skills/failure-autopsy-and-healing/SKILL.md +2 -2
- package/.agents/skills/failure-autopsy-and-healing/scripts/triage_failures.sh +6 -21
- package/.agents/skills/katalon-cli-runner/SKILL.md +1 -1
- package/.agents/skills/mipro-prompt-compiler/SKILL.md +59 -0
- package/.agents/skills/mipro-prompt-compiler/examples/compiled_module_sample.json +22 -0
- package/.agents/skills/qa-aci-interface/SKILL.md +67 -0
- package/.agents/skills/qa-aci-interface/examples/surgical_edit_sample.json +12 -0
- package/.agents/skills/qa-memory-os/SKILL.md +80 -0
- package/.agents/skills/qa-memory-os/examples/archival_memory_sample.json +21 -0
- package/.agents/skills/review-spec/SKILL.md +60 -0
- package/.agents/skills/review-spec/examples/sample_spec_review.md +33 -0
- package/.agents/skills/synthetic-test-data-generator/SKILL.md +2 -2
- package/.agents/skills/token-and-context-optimizer/references/context_engineering_guide.md +49 -0
- package/.agents/skills/tot-exploratory-testing/SKILL.md +59 -0
- package/.agents/skills/tot-exploratory-testing/examples/tot_graph_sample.json +35 -0
- package/.agents/skills/wcag-audit-patterns/SKILL.md +2 -2
- package/.agents/skills/web-agent-gym-benchmarking/SKILL.md +62 -0
- package/.agents/skills/web-agent-gym-benchmarking/examples/benchmark_report_sample.json +22 -0
- package/AGENTS.md +6 -1
- package/CLAUDE.md +1 -1
- package/NOTICE +40 -4
- package/README.en.md +27 -11
- package/README.md +28 -12
- package/docs/ARCHITECTURE.md +39 -11
- package/docs/adopted-ecosystem-tools.md +9 -0
- package/docs/research/EXTERNAL_AGENT_SKILLS_SURVEY.md +1062 -0
- package/docs/research/agent_skill_training_analysis.md +1357 -0
- package/package.json +1 -1
- package/skills_manifest.json +88 -7
- package/synapse.config.json +3 -2
- package/templates/ai_bridges/CLAUDE.md +1 -1
- package/templates/welcome.html.tmpl +34 -25
- package/tools/TOOLS_RUNBOOK.md +130 -1
- package/tools/__init__.py +1 -1
- package/tools/_openapi_parser.py +158 -0
- package/tools/agentless_patcher.py +417 -0
- package/tools/analyze_project.py +9 -6
- package/tools/auto_heal.py +29 -0
- package/tools/auto_heal_reflexion.py +172 -0
- package/tools/autogen_qa_broker.py +318 -0
- package/tools/browser_helpers.js +57 -2
- package/tools/browsergym_eval.py +237 -0
- package/tools/ci_healing_hook.py +1 -1
- package/tools/cli.py +3 -3
- package/tools/dashboard_generator.py +29 -13
- package/tools/db_sandbox.py +4 -0
- package/tools/dspy_qa_signatures.py +357 -0
- package/tools/heal_pr_automator.py +25 -0
- package/tools/interview_story.py +117 -0
- package/tools/lint_ai_docs.py +1 -1
- package/tools/mcp_excel_server.py +1 -1
- package/tools/mcp_server.py +1 -1
- package/tools/mipro_assertion_tuner.py +207 -0
- package/tools/pinpoint_failure.py +12 -0
- package/tools/project_state.py +4 -0
- package/tools/qa_event_stream.py +336 -0
- package/tools/reversible_sandbox.py +161 -0
- package/tools/safety_firewall.py +28 -0
- package/tools/skill_collision_guard.py +231 -0
- package/tools/skill_vault.py +343 -0
- package/tools/swe_qa_aci.py +369 -0
- package/tools/synapse_qa_bench.py +190 -0
- package/tools/sync_agent_rules.py +1 -1
- package/tools/test_memory_os.py +409 -0
- package/tools/textgrad_mutation_optimizer.py +234 -0
- package/tools/tot_exploratory_tester.py +240 -0
- package/tools/update_sentinel.py +3 -3
- package/tools/verify_all.py +35 -4
- package/tools/watch_mode.py +1 -1
- package/tools/welcome_page.py +2 -2
|
@@ -0,0 +1,1357 @@
|
|
|
1
|
+
# Agent Skill Training, Optimization & Synthesis: Comparative Analysis and Synapse-Hub Architectural Blueprint
|
|
2
|
+
|
|
3
|
+
> **Author**: Thế Vị (@TeeVee998) & Synapse-Hub Core Architecture Team
|
|
4
|
+
> **Repository**: [https://github.com/TeeVee998/Synapse-Hub](https://github.com/TeeVee998/Synapse-Hub)
|
|
5
|
+
> **Document Status**: Production Research Deliverable & Architecture Specification
|
|
6
|
+
> **Target Version**: Synapse-Hub v2.6.0+
|
|
7
|
+
> **Date**: November 2026
|
|
8
|
+
> **License**: MIT (Attribution Required)
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## Table of Contents
|
|
13
|
+
|
|
14
|
+
1. [Executive Summary & Research Context](#1-executive-summary--research-context)
|
|
15
|
+
- 1.1 Problem Statement & Background
|
|
16
|
+
- 1.2 Objectives & Scope
|
|
17
|
+
- 1.3 Research Methodology & Acquisition Verification
|
|
18
|
+
2. [In-Depth Analysis of 10 Agent Skill Training & Optimization Repositories](#2-in-depth-analysis-of-10-agent-skill-training--optimization-repositories)
|
|
19
|
+
- 2.1 MineDojo/Voyager (Lifelong Embodied Learning & Vectorized Skill Libraries)
|
|
20
|
+
- 2.2 noahshinn/Reflexion (Verbal Reinforcement Learning & Episodic Self-Reflection)
|
|
21
|
+
- 2.3 stanfordnlp/dspy (Declarative LM Programming & Metric-Driven Teleprompters)
|
|
22
|
+
- 2.4 intertwine/dspy-agent-skills (Production Agent Skill Packaging for DSPy 3.2.x)
|
|
23
|
+
- 2.5 zou-group/textgrad (Automatic Differentiation via Textual Gradients)
|
|
24
|
+
- 2.6 VoltAgent/awesome-agent-skills (Universal Skill Taxonomies & Standards)
|
|
25
|
+
- 2.7 addyosmani/agent-skills (SDLC Lifecycle Disciplines, 3-Tier Evals & Anti-Rationalization)
|
|
26
|
+
- 2.8 mattpocock/skills (Grilling Primitive, Ubiquitous Glossaries & Wayfinder State Machines)
|
|
27
|
+
- 2.9 OpenBMB/ToolBench (Depth-First Search Decision Trees & Tool Retrieval)
|
|
28
|
+
- 2.10 ShishirPatil/gorilla (BFCL Function Calling Benchmark & GoEx Reversible Execution)
|
|
29
|
+
3. [Comprehensive 10-Repository Comparative Matrix](#3-comprehensive-10-repository-comparative-matrix)
|
|
30
|
+
- 3.1 Multi-Dimensional Comparative Evaluation Table
|
|
31
|
+
- 3.2 Quantitative Capability & QA Adaptability Scoring
|
|
32
|
+
4. [Cross-Cutting Analysis of 5 Core Training & Optimization Paradigms](#4-cross-cutting-analysis-of-5-core-training--optimization-paradigms)
|
|
33
|
+
- 4.1 Paradigm 1: Curriculum Learning & Lifelong Skill Libraries
|
|
34
|
+
- 4.2 Paradigm 2: Verbal Reinforcement Learning & Multi-Trial Reflection Loops
|
|
35
|
+
- 4.3 Paradigm 3: Declarative LM Programming & Teleprompter Prompt Compilation
|
|
36
|
+
- 4.4 Paradigm 4: Automatic Differentiation & Textual Gradients
|
|
37
|
+
- 4.5 Paradigm 5: Large-Scale Tool-Use, Function Calling & Sandboxed Execution
|
|
38
|
+
5. [Synapse-Hub Current Architecture Baseline](#5-synapse-hub-current-architecture-baseline)
|
|
39
|
+
- 5.1 Hub-and-Spoke Decoupled Operating Model
|
|
40
|
+
- 5.2 Clean POM, Flat Architecture & Universal IR
|
|
41
|
+
- 5.3 Playwright Live Interaction & Fast-Loop Discovery
|
|
42
|
+
- 5.4 Verification Discipline & The 9 Quality Gates DoD
|
|
43
|
+
- 5.5 Dual-Storage SQLite Episodic & Semantic Memory Cache
|
|
44
|
+
- 5.6 6-Agent Swarm System & Single Front-Door FSM
|
|
45
|
+
- 5.7 The 6 Pillars of Context Engineering & Token Optimization
|
|
46
|
+
6. [Architectural Integration Blueprint for Synapse-Hub](#6-architectural-integration-blueprint-for-synapse-hub)
|
|
47
|
+
- 6.1 Target Architecture Blueprint & System Topology (Mermaid)
|
|
48
|
+
- 6.2 Subsystem 1: SkillVault Lifelong POM Library (Voyager Pattern)
|
|
49
|
+
- 6.3 Subsystem 2: Multi-Trial Episodic Reflexion Loop in `auto_heal.py` (Reflexion Pattern)
|
|
50
|
+
- 6.4 Subsystem 3: Declarative QA Signature Compilation Engine (DSPy Pattern)
|
|
51
|
+
- 6.5 Subsystem 4: Mutant-Directed Textual Loss Optimizer (TextGrad Pattern)
|
|
52
|
+
- 6.6 Subsystem 5: Synapse-QA-Bench & Reversible Execution Sandbox (Gorilla & ToolBench Pattern)
|
|
53
|
+
- 6.7 Subsystem 6: Modern Packaging, Tier-2 Collision Guard & Grilling Primitive (AddyOsmani & MattPocock Pattern)
|
|
54
|
+
- 6.8 End-to-End Execution Sequence Flow (Mermaid)
|
|
55
|
+
7. [Actionable Prioritized Integration Roadmap & Trade-Off Analysis](#7-actionable-prioritized-integration-roadmap--trade-off-analysis)
|
|
56
|
+
- 7.1 Prioritized Phased Implementation Plan (P0, P1, P2)
|
|
57
|
+
- 7.2 Technical Trade-Off & Cost-Benefit Analysis
|
|
58
|
+
- 7.3 Risk Matrix & Mitigation Strategies
|
|
59
|
+
- 7.4 Definition of Done (DoD) & Acceptance Sign-Off
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## 1. Executive Summary & Research Context
|
|
64
|
+
|
|
65
|
+
### 1.1 Problem Statement & Background
|
|
66
|
+
|
|
67
|
+
Autonomous AI software engineering agents have rapidly transitioned from prototype chat interfaces into complex, multi-agent developer workflows capable of authoring code, triaging test failures, and deploying applications. However, modern agent implementations frequently suffer from four critical systemic failure modes:
|
|
68
|
+
|
|
69
|
+
1. **Brittle Prompt Fragility & Hallucination**: Prompts constructed from unstructured natural language strings regress when underlying Large Language Models (LLMs) update their weights, change system instructions, or encounter slight perturbations in input data.
|
|
70
|
+
2. **Amnesiac Repetition & Blind Retries**: When an agent attempts an action that fails due to environmental constraints (e.g., stale DOM locators, transient network timeouts, or schema mismatches), naive agent loops repeatedly execute identical or minor syntactic variations of the failing patch without understanding the underlying root cause.
|
|
71
|
+
3. **Context Bloat & Token Degradation ("Context Rot")**: Dumping thousands of raw lines of CLI execution logs, uncompressed HTML DOM trees, and monolithic source files into the context window causes catastrophic degradation of LLM reasoning capacity (the well-documented "Lost-in-the-Middle" phenomenon) and inflates operational API costs.
|
|
72
|
+
4. **Unverified Optimism ("Fake Passes")**: Without deterministic verification harnesses, AI agents routinely relax assertions, swallow exceptions, or introduce empty catch blocks (`try ... catch {}`) to simulate successful task completion, destroying enterprise software reliability.
|
|
73
|
+
|
|
74
|
+
In response to these challenges, the academic and open-source communities have developed diverse methodologies for **agent skill training, prompt compilation, episodic self-reflection, and tool benchmarking**. These range from lifelong curriculum exploration (Voyager) and verbal reinforcement learning (Reflexion) to declarative pipeline compilation (DSPy), backpropagation through text (TextGrad), and function-calling evaluation suites (Gorilla BFCL, ToolBench).
|
|
75
|
+
|
|
76
|
+
### 1.2 Objectives & Scope
|
|
77
|
+
|
|
78
|
+
This research deliverable executes a deep-dive investigation into **10 seminal repositories** spanning agent skill training, optimization, and packaging. The primary objective is to evaluate how the underlying algorithms, data structures, and architectural patterns of these frameworks can be synthesized into **Synapse-Hub**, an enterprise-grade automated QA and testing intelligence platform.
|
|
79
|
+
|
|
80
|
+
The investigation focuses specifically on:
|
|
81
|
+
- Dissecting the core mechanisms, modularity, strengths, weaknesses, token economics, and QA relevance of all 10 repositories.
|
|
82
|
+
- Conducting a cross-cutting synthesis across 5 fundamental paradigms of agent optimization.
|
|
83
|
+
- Documenting Synapse-Hub's existing architectural baseline as the foundation for integration.
|
|
84
|
+
- Formulating an end-to-end Architectural Integration Blueprint with concrete system topology diagrams, component-level designs, and code-level interface specifications.
|
|
85
|
+
- Delivering a phased, prioritized implementation roadmap (P0, P1, P2) with thorough trade-off and risk analyses.
|
|
86
|
+
|
|
87
|
+
### 1.3 Research Methodology & Acquisition Verification
|
|
88
|
+
|
|
89
|
+
To guarantee rigorous empirical analysis rather than theoretical speculation, all 10 repositories were acquired via shallow cloning (`git clone --depth 1`) into an isolated workspace directory (`external_repos/`). The workspace was protected using Git exclude rules (`.git/info/exclude`) ensuring that external clones remained completely invisible to Git without modifying tracked repository files.
|
|
90
|
+
|
|
91
|
+
The 10 acquired repositories and their verified checkout references are:
|
|
92
|
+
|
|
93
|
+
```text
|
|
94
|
+
external_repos/
|
|
95
|
+
├── Voyager/ [MineDojo/Voyager, Commit: 55e45a8, NeurIPS 2023]
|
|
96
|
+
├── Reflexion/ [noahshinn/reflexion (canonical), Commit: 218cf0e, NeurIPS 2023]
|
|
97
|
+
├── dspy/ [stanfordnlp/dspy, Commit: c941671, v2.5/v2.6/v3.0]
|
|
98
|
+
├── dspy-agent-skills/ [intertwine/dspy-agent-skills, Commit: 623dca0, DSPy 3.2.x]
|
|
99
|
+
├── textgrad/ [zou-group/textgrad, Commit: 75e912e, Nature 2025]
|
|
100
|
+
├── awesome-agent-skills/ [VoltAgent/awesome-agent-skills, Commit: 459d242, 1,497+ Skills]
|
|
101
|
+
├── agent-skills/ [addyosmani/agent-skills, Commit: 301b02b, 25 Skills, 3-Tier Evals]
|
|
102
|
+
├── mattpocock-skills/ [mattpocock/skills, Commit: 4ae2317, Grilling Primitive]
|
|
103
|
+
├── ToolBench/ [OpenBMB/ToolBench, Commit: d56fdd8, 16,464 APIs, DFSDT]
|
|
104
|
+
└── gorilla/ [ShishirPatil/gorilla, Commit: 6ea5797, BFCL v1-v4, GoEx]
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Every code reference, file path, and algorithmic description in Section 2 is verified directly against the cloned codebases on disk.
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## 2. In-Depth Analysis of 10 Agent Skill Training & Optimization Repositories
|
|
112
|
+
|
|
113
|
+
### 2.1 MineDojo/Voyager (Lifelong Embodied Learning & Vectorized Skill Libraries)
|
|
114
|
+
|
|
115
|
+
- **Repository**: `MineDojo/Voyager` (`https://github.com/MineDojo/Voyager`)
|
|
116
|
+
- **Key Files & Entrypoints**: `voyager/voyager.py`, `voyager/agents/action.py`, `voyager/agents/critic.py`, `voyager/agents/curriculum.py`, `voyager/agents/skill.py`, `voyager/control_primitives/`.
|
|
117
|
+
- **Publication**: NeurIPS 2023 (*Voyager: An Open-Ended Embodied Agent with Large Language Models* by Wang et al., Stanford, NVIDIA, Caltech, UT Austin).
|
|
118
|
+
|
|
119
|
+
```
|
|
120
|
+
+-----------------------------+
|
|
121
|
+
| CurriculumAgent |
|
|
122
|
+
| (Proposes Exploration Goal) |
|
|
123
|
+
+--------------+--------------+
|
|
124
|
+
|
|
|
125
|
+
v
|
|
126
|
+
+------------------+ Task & Context +----------------------+
|
|
127
|
+
| SkillManager |---------------->| ActionAgent |
|
|
128
|
+
| (Vector Storage: | Top-K Programs | (Synthesizes JS Code)|
|
|
129
|
+
| ChromaDB + Embed)|<----------------+----------+-----------+
|
|
130
|
+
+------------------+ Verified Code |
|
|
131
|
+
^ v
|
|
132
|
+
| Task Success +--------------------+
|
|
133
|
+
+-----------------------------| CriticAgent |
|
|
134
|
+
| (Verifies Outcome) |
|
|
135
|
+
+---------+----------+
|
|
136
|
+
| Runtime Failure
|
|
137
|
+
v
|
|
138
|
+
Iterative Error Feedback
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
#### A. Purpose & Core Concept
|
|
142
|
+
Voyager is the first LLM-powered embodied lifelong learning agent capable of continuous exploration, novel task discovery, and cumulative skill acquisition in open-ended environments (Minecraft) without human supervision. Voyager demonstrates that an agent can autonomously expand its own procedural knowledge by treating synthesized code as executable skills stored in a vector-indexed library.
|
|
143
|
+
|
|
144
|
+
#### B. Key Architectural Mechanisms
|
|
145
|
+
1. **Automatic Curriculum Engine (`CurriculumAgent`)**:
|
|
146
|
+
- `voyager/agents/curriculum.py` continuously evaluates the agent's current inventory, nearby environment entities, biome context, and completed task history.
|
|
147
|
+
- Using structured prompting with GPT-4, it autonomously proposes next-step goals that optimize for the "Zone of Proximal Development": tasks that are neither trivially accomplished nor impossibly difficult given current assets.
|
|
148
|
+
2. **Iterative Code Synthesis & Compiler Feedback Loop (`ActionAgent`)**:
|
|
149
|
+
- `voyager/agents/action.py` generates executable JavaScript programs utilizing the Mineflayer control API.
|
|
150
|
+
- If program execution encounters JavaScript syntax errors, runtime exceptions, or API timeouts in the Node.js sub-process, the execution trace and console errors are piped back into the agent's prompt context.
|
|
151
|
+
- The agent iteratively refines the code in-place for up to `action_agent_task_max_retries = 4` rounds before declaring failure.
|
|
152
|
+
3. **Environment State Verification Critic (`CriticAgent`)**:
|
|
153
|
+
- `voyager/agents/critic.py` solves the hallucination problem in code generation by acting as an independent judge.
|
|
154
|
+
- Rather than trusting the LLM's internal conviction that its code succeeded, the critic queries the underlying environment state before and after execution (e.g., inventory item count delta: `iron_ingot: +3`). Only if the empirical delta satisfies the target goal is the task validated.
|
|
155
|
+
4. **Vector-Indexed Skill Library (`SkillManager`)**:
|
|
156
|
+
- `voyager/agents/skill.py` converts verified code into permanent assets.
|
|
157
|
+
- When a program passes Critic verification, `SkillManager` prompts an LLM to generate a concise docstring describing the program's utility and pre-conditions.
|
|
158
|
+
- The docstring is vectorized using `OpenAIEmbeddings` and stored in a ChromaDB vector store.
|
|
159
|
+
- When new tasks are assigned, `SkillManager` performs a semantic search (`top_k = 5`) to retrieve relevant existing skills, injecting them into the `ActionAgent` prompt context as composable primitives.
|
|
160
|
+
|
|
161
|
+
#### C. Modularity & Codebase Structure
|
|
162
|
+
- **Decoupled Agent Subsystems**: The 4 agents (`CurriculumAgent`, `ActionAgent`, `CriticAgent`, `SkillManager`) communicate strictly via structured JSON and function interfaces orchestrated by `voyager.py`.
|
|
163
|
+
- **Environment Abstraction**: `voyager/env/` encapsulates the Mineflayer Node.js bridge, isolating the cognitive agent logic from low-level IPC sockets.
|
|
164
|
+
|
|
165
|
+
#### D. Pros & Cons
|
|
166
|
+
- **Strengths**:
|
|
167
|
+
- *Lifelong Continual Learning*: Procedural knowledge grows monotonically; the agent becomes exponentially faster at solving complex tasks as its skill library accumulates.
|
|
168
|
+
- *Compositional Hierarchy*: High-level complex tasks naturally compose simpler verified sub-routines (e.g., `craftWoodenPickaxe()` calls `mineWood()` and `craftPlanks()`).
|
|
169
|
+
- *Truth Anchor via State Verification*: The Critic eliminates false-positive success claims by grounding verification in environment state.
|
|
170
|
+
- **Weaknesses & Limitations**:
|
|
171
|
+
- *Token & Compute Intensity*: Each rollout involves multiple high-reasoning LLM calls (task proposal, action generation, multi-turn error correction, critic inspection, docstring generation, and embedding).
|
|
172
|
+
- *Brittle Textual Code Parsing*: Heavily reliant on regex and markdown code-block extractors (````javascript ... ````).
|
|
173
|
+
|
|
174
|
+
#### E. Token & Compute Efficiency
|
|
175
|
+
- **Training / Discovery Phase**: High token burden. An exploratory task requires ~8,000 to 25,000 tokens across retries, critic evaluations, and skill indexing.
|
|
176
|
+
- **Inference / Execution Phase**: Moderate. Once a skill library is populated, tasks are resolved by retrieving top-5 skills (~1,500 tokens), bypassing exploratory search.
|
|
177
|
+
|
|
178
|
+
#### F. Relevance & Adaptability to Enterprise QA Automation
|
|
179
|
+
- **Score: 9.2 / 10**
|
|
180
|
+
- **Direct QA Mapping**: Voyager's Skill Library maps 1:1 to an autonomous **Page Object Model (POM) Library**. In web automation, atomic UI interactions (e.g., `loginWithSSO()`, `selectDateRange()`, `fillDynamicGrid()`) are verified against live browser DOM, annotated with semantic descriptions, and indexed into a database for reuse across test cases.
|
|
181
|
+
|
|
182
|
+
---
|
|
183
|
+
|
|
184
|
+
### 2.2 noahshinn/Reflexion (Verbal Reinforcement Learning & Episodic Self-Reflection)
|
|
185
|
+
|
|
186
|
+
- **Repository**: `noahshinn/reflexion` (`https://github.com/noahshinn/reflexion`, canonical; referenced historically as `noahshalo/Reflexion`).
|
|
187
|
+
- **Key Files & Entrypoints**: `programming_runs/reflexion.py`, `programming_runs/generators/py_generate.py`, `programming_runs/executors/`, `hotpotqa_runs/`, `alfworld_runs/`.
|
|
188
|
+
- **Publication**: NeurIPS 2023 (*Reflexion: Language Agents with Verbal Reinforcement Learning* by Shinn et al., Northeastern University, MIT, Princeton).
|
|
189
|
+
|
|
190
|
+
```
|
|
191
|
+
+--------------------------------------------------------------------+
|
|
192
|
+
| Reflexion Architecture |
|
|
193
|
+
| |
|
|
194
|
+
| +-------------+ Trajectory +-------------------+ |
|
|
195
|
+
| | Actor |----------------------->| Evaluator | |
|
|
196
|
+
| | (Generator) | | (Unit Tests / Py) | |
|
|
197
|
+
| +------+------+ +---------+---------+ |
|
|
198
|
+
| ^ | |
|
|
199
|
+
| | Context: Code + Feedback + Refl | Feedback: |
|
|
200
|
+
| | | Stacktrace |
|
|
201
|
+
| | v |
|
|
202
|
+
| +------+------+ Verbal Reflection +---------+---------+ |
|
|
203
|
+
| | Episodic |<-----------------------| Self-Reflection | |
|
|
204
|
+
| | Memory | "Failed because loop | Generator | |
|
|
205
|
+
| +-------------+ index was off-by-one"| (Verbal Critique) | |
|
|
206
|
+
| +-------------------+ |
|
|
207
|
+
+--------------------------------------------------------------------+
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
#### A. Purpose & Core Concept
|
|
211
|
+
Reflexion introduces Verbal Reinforcement Learning, a paradigm where agents learn from trial and error without computing mathematical gradients or fine-tuning model weights. By converting scalar reward signals and unit test execution stacktraces into descriptive, natural language "self-reflections", the agent stores its reasoning missteps in an episodic memory buffer to guide subsequent trials.
|
|
212
|
+
|
|
213
|
+
#### B. Key Architectural Mechanisms
|
|
214
|
+
1. **Actor-Evaluator-Reflection Triad**:
|
|
215
|
+
- `Actor` (`py_generate.py`): Generates candidate code implementations or reasoning trajectories based on task prompt and memory context.
|
|
216
|
+
- `Evaluator` (`executors/`): Executes the generated code in a sandboxed sub-process against unit test suites. Outputs binary pass/fail and captures verbatim standard error/stacktraces.
|
|
217
|
+
- `Self-Reflection Generator`: When a trial fails, the reflection module prompts the LLM with the task, the failing code, and the test execution feedback. The model is explicitly commanded: *"Write 2-3 sentences explaining why the implementation failed and how to fix it. Do NOT output code."*
|
|
218
|
+
2. **Episodic Reflection Buffer**:
|
|
219
|
+
- The agent maintains an in-memory buffer of reflections across trials:
|
|
220
|
+
$$\mathcal{M} = [r_1, r_2, \dots, r_t]$$
|
|
221
|
+
- On trial $t+1$, the prompt context includes: the original specification, previous failing code, test failure output, and the accumulated reflections $\mathcal{M}$.
|
|
222
|
+
3. **Loop Termination Criteria**:
|
|
223
|
+
- The loop terminates immediately upon all unit tests passing (`is_passing == True`), achieving optimal token economy, or halts when reaching a maximum iteration threshold (`max_iters = 4`).
|
|
224
|
+
|
|
225
|
+
#### C. Modularity & Codebase Structure
|
|
226
|
+
- `programming_runs/`: Benchmarking harness across HumanEval and MBPP.
|
|
227
|
+
- `generators/factory.py`: Pluggable model generator abstraction supporting OpenAI, Anthropic, and local LLMs.
|
|
228
|
+
- `executors/`: Language-specific sandboxes executing Python and Rust in isolated processes with strict timeout limits.
|
|
229
|
+
|
|
230
|
+
#### D. Pros & Cons
|
|
231
|
+
- **Strengths**:
|
|
232
|
+
- *Dramatic First-Trial Accuracy Boost*: Demonstrates substantial accuracy gains (e.g., HumanEval initial accuracy leaps from 67% on baseline GPT-4 to 91% on Reflexion).
|
|
233
|
+
- *Zero Weight Modifications*: Requires no fine-tuning datasets, GPU clusters, or LoRA adapters.
|
|
234
|
+
- *Explainable Debug Trajectory*: Human engineers can inspect the episodic memory to understand the exact cognitive chain that healed the bug.
|
|
235
|
+
- **Weaknesses & Limitations**:
|
|
236
|
+
- *Hallucinated Reflections*: If test error messages are opaque (e.g., `AssertionError: assert False`), the reflection engine can hallucinate incorrect diagnoses.
|
|
237
|
+
- *Context Window Expansion*: Unbounded accumulation of reflections expands prompt context, risking attention dilution.
|
|
238
|
+
|
|
239
|
+
#### E. Token & Compute Efficiency
|
|
240
|
+
- **Trial Overhead**: Exceptionally lightweight compared to parameter optimization. Each failed iteration adds exactly 1 brief reflection completion (~200–350 tokens) and 1 regeneration call (~600–1,000 tokens).
|
|
241
|
+
- **Execution Cost**: Overall token consumption scales linearly with the number of failed attempts; successful runs terminate on trial 1 with zero reflection overhead.
|
|
242
|
+
|
|
243
|
+
#### F. Relevance & Adaptability to Enterprise QA Automation
|
|
244
|
+
- **Score: 9.8 / 10**
|
|
245
|
+
- **Direct QA Mapping**: The Reflexion loop is the exact architectural primitive required for **Automated Test Failure Healing (`auto_heal.py`)** and **Failure Autopsy Analysis**. When headless CI tests fail, the stacktrace and JUnit XML should trigger a verbal reflection loop that diagnoses the root cause before attempting locator updates or assertion patches.
|
|
246
|
+
|
|
247
|
+
---
|
|
248
|
+
|
|
249
|
+
### 2.3 stanfordnlp/dspy (Declarative LM Programming & Metric-Driven Teleprompters)
|
|
250
|
+
|
|
251
|
+
- **Repository**: `stanfordnlp/dspy` (`https://github.com/stanfordnlp/dspy`)
|
|
252
|
+
- **Key Files & Entrypoints**: `dspy/signatures/signature.py`, `dspy/predict/predict.py` (and `predict/refine.py`), `dspy/teleprompt/bootstrap.py`, `dspy/teleprompt/mipro_optimizer_v2.py`, `dspy/teleprompt/gepa/`, `dspy/primitives/`.
|
|
253
|
+
- **Publication**: ICLR 2024 (*DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines* by Khattab et al., Stanford University).
|
|
254
|
+
|
|
255
|
+
```
|
|
256
|
+
+-----------------------------------------------------------------+
|
|
257
|
+
| DSPy Compilation Flow |
|
|
258
|
+
| |
|
|
259
|
+
| +-----------------------+ +----------------------+ |
|
|
260
|
+
| | Declarative Signature | | Metric Function | |
|
|
261
|
+
| | class QA(Signature): | | (DoD Quality Gates, | |
|
|
262
|
+
| | code -> test_script | | Mutation Score >=80)| |
|
|
263
|
+
| +-----------+-----------+ +----------+-----------+ |
|
|
264
|
+
| | | |
|
|
265
|
+
| +----------------+----------------+ |
|
|
266
|
+
| | |
|
|
267
|
+
| v |
|
|
268
|
+
| +-------------------------+ |
|
|
269
|
+
| | Teleprompter Optimizer | |
|
|
270
|
+
| | (MIPROv2 / GEPA / Boot) | |
|
|
271
|
+
| +------------+------------+ |
|
|
272
|
+
| | |
|
|
273
|
+
| v |
|
|
274
|
+
| +-------------------------+ |
|
|
275
|
+
| | Compiled Prompt Program | |
|
|
276
|
+
| | (Frozen High-Performance| |
|
|
277
|
+
| | Zero-Token Runtime) | |
|
|
278
|
+
| +-------------------------+ |
|
|
279
|
+
+-----------------------------------------------------------------+
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
#### A. Purpose & Core Concept
|
|
283
|
+
DSPy ("Declarative Self-improving Language Programs, Pythonically") replaces brittle, hand-crafted prompt strings with structured, declarative Python signatures and modular pipeline abstractions analogous to PyTorch (`torch.nn.Module`). DSPy introduces algorithmic **Teleprompters (Optimizers)** that compile prompt pipelines against quantitative evaluation metrics, automatically synthesizing optimal system instructions and selecting few-shot demonstrations.
|
|
284
|
+
|
|
285
|
+
#### B. Key Architectural Mechanisms
|
|
286
|
+
1. **Declarative Signatures (`dspy.Signature`)**:
|
|
287
|
+
- Decouples task semantics from prompt formatting.
|
|
288
|
+
- Example:
|
|
289
|
+
```python
|
|
290
|
+
class GenerateCleanPOM(dspy.Signature):
|
|
291
|
+
"""Synthesize a Clean POM Page Class adhering to Flat Architecture."""
|
|
292
|
+
dom_aom_snapshot = dspy.InputField(desc="Compact AOM tree with roles and testids")
|
|
293
|
+
target_workflows = dspy.InputField(desc="Business user interactions to encapsulate")
|
|
294
|
+
page_class_code = dspy.OutputField(desc="Executable BasePage subclass with zero raw XPaths")
|
|
295
|
+
```
|
|
296
|
+
2. **Modular Predictor Primitives (`dspy.Predict`, `dspy.ChainOfThought`, `dspy.ReAct`)**:
|
|
297
|
+
- Standardizes reasoning patterns. Wrapping a signature in `dspy.ChainOfThought(GenerateCleanPOM)` automatically injects step-by-step reasoning tokens without modifying the signature definition.
|
|
298
|
+
3. **Teleprompters & Optimization Algorithms**:
|
|
299
|
+
- `BootstrapFewShot`: Automatically runs unlabelled training instances through the pipeline, verifies which traces satisfy the metric, and compiles the passing traces into few-shot exemplars.
|
|
300
|
+
- `COPRO` & `MIPROv2`: Uses coordinate ascent and Bayesian optimization (via Optuna) to search the joint space of natural language instructions and demonstration sets.
|
|
301
|
+
- `GEPA` (Reflective Prompt Evolution, 2025): Uses LLM-generated verbal critiques of failure modes to guide the evolutionary mutation of prompt instructions, outperforming standard reinforcement learning.
|
|
302
|
+
4. **Computational Assertions, Suggestions & Evolution to `dspy.Refine` (`dspy.Assert`, `dspy.Suggest`, `dspy.Refine`)**:
|
|
303
|
+
- Allows hard deterministic constraints to be embedded directly into LM pipelines:
|
|
304
|
+
`dspy.Assert(lambda out: "xpath" not in out.lower(), "Raw XPaths are strictly forbidden in Clean POM!")`
|
|
305
|
+
- If an assertion fails during execution, DSPy dynamically backtracks and re-queries the model, injecting the assertion error into the prompt.
|
|
306
|
+
- *Modern Evolution*: In modern DSPy v2.5+/v2.6+ (`c941671`), legacy standalone assertion primitives in `dspy/primitives/` have evolved into module-level retry primitives and reward-guided refinement via `dspy.Refine` (`dspy/predict/refine.py`), where multi-round natural language feedback and reward functions guide self-correction without brittle backtracking hacks.
|
|
307
|
+
|
|
308
|
+
#### C. Modularity & Codebase Structure
|
|
309
|
+
- `dspy/signatures/`: Schema validation, type hints, and field specifications.
|
|
310
|
+
- `dspy/primitives/`: Core execution layers, backtracking interpreters, and sandboxing.
|
|
311
|
+
- `dspy/teleprompt/`: Comprehensive suite of 20+ algorithmic prompt optimizers.
|
|
312
|
+
- `dspy/adapters/`: Multi-provider adapters for OpenAI, Anthropic, Cohere, Bedrock, and local models.
|
|
313
|
+
|
|
314
|
+
#### D. Pros & Cons
|
|
315
|
+
- **Strengths**:
|
|
316
|
+
- *Engineering Rigor*: Treats prompts as software parameters compiled against objective code metrics.
|
|
317
|
+
- *Model Portability*: Compiles distinct, optimal prompts for different models (e.g., generating distinct instruction variants for GPT-4o vs. LLaMA-3-8B).
|
|
318
|
+
- *Zero-Cost Runtime*: Once compiled, the pipeline is serialized to a JSON artifact containing frozen instructions and demonstrations, requiring zero prompt search tokens at runtime.
|
|
319
|
+
- **Weaknesses & Limitations**:
|
|
320
|
+
- *Dataset Requirement*: Optimization requires a labeled or unlabeled training dataset (typically 20–100 representative examples).
|
|
321
|
+
- *Compilation Token Cost*: Running teleprompters (especially MIPROv2) requires hundreds of candidate evaluation calls.
|
|
322
|
+
|
|
323
|
+
#### E. Token & Compute Efficiency
|
|
324
|
+
- **Compilation Phase**: Heavy token expenditure (e.g., 50,000–300,000 tokens during offline Bayesian search).
|
|
325
|
+
- **Runtime Inference Phase**: Maximum efficiency. The compiled prompt replaces sprawling conversational back-and-forths with an ultra-compact, highly calibrated system prompt.
|
|
326
|
+
|
|
327
|
+
#### F. Relevance & Adaptability to Enterprise QA Automation
|
|
328
|
+
- **Score: 9.9 / 10**
|
|
329
|
+
- **Direct QA Mapping**: Synapse-Hub's **9 Quality Gates DoD (`tools/verify_all.py`)** and **Micro-Mutation Testing (`tools/micro_mutator.py`)** provide the ideal deterministic scalar metric required by DSPy teleprompters. DSPy can autonomously compile and harden prompt instructions for `pom-crafter` and `lam-script-generator`.
|
|
330
|
+
|
|
331
|
+
---
|
|
332
|
+
|
|
333
|
+
### 2.4 intertwine/dspy-agent-skills (Production Agent Skill Packaging for DSPy 3.2.x)
|
|
334
|
+
|
|
335
|
+
- **Repository**: `intertwine/dspy-agent-skills` (`https://github.com/intertwine/dspy-agent-skills`)
|
|
336
|
+
- **Key Files & Entrypoints**: `README.md`, `skills/`, `examples/`, `tests/test_skill_metadata.py`, `scripts/check_dspy_surface.py`.
|
|
337
|
+
- **Packaging Standard**: Conforms to the emerging `agentskills.io` standard (compatible with Claude Code, Codex, and Antigravity).
|
|
338
|
+
|
|
339
|
+
#### A. Purpose & Core Concept
|
|
340
|
+
`dspy-agent-skills` provides production-ready agent skills that package DSPy 3.2.x capabilities directly into autonomous coding workflows. It operationalizes DSPy within developer IDEs and coding agents, demonstrating how to package complex prompt compilation routines into portable, spec-compliant skills.
|
|
341
|
+
|
|
342
|
+
#### B. Key Architectural Mechanisms
|
|
343
|
+
1. **Progressive Disclosure Directory Structure**:
|
|
344
|
+
- Standardizes the skill anatomy across 5 specialized domains:
|
|
345
|
+
- `dspy-fundamentals`: Signatures, modules, and serialization workflows.
|
|
346
|
+
- `dspy-evaluation-harness`: Metrics, splits, and `dspy.Evaluate` runners.
|
|
347
|
+
- `dspy-gepa-optimizer`: Reflective prompt evolution pipelines.
|
|
348
|
+
- `dspy-rlm-module`: Long-context recursive codebase exploration.
|
|
349
|
+
- `dspy-advanced-workflow`: End-to-end build-evaluate-optimize-deploy pipelines.
|
|
350
|
+
- Each skill directory encapsulates:
|
|
351
|
+
- `SKILL.md`: Ultra-compact entrypoint (< 150 lines) with YAML frontmatter.
|
|
352
|
+
- `reference.md`: In-depth documentation loaded on demand.
|
|
353
|
+
- `example_*.py`: Executable code demonstrations with `--dry-run` flags.
|
|
354
|
+
2. **Offline Verification Test Suite (`tests/`)**:
|
|
355
|
+
- `tests/test_skill_metadata.py`: Validates YAML frontmatter syntax, field naming (`name`, `description`, optional `when_to_use`), and strictly enforces rejection of legacy/unsupported fields (`triggers`, `version`, `dspy-compatibility`, `dspy-version`).
|
|
356
|
+
- `check_dspy_surface.py`: Validates that all referenced Python APIs match the installed DSPy release, preventing API hallucination.
|
|
357
|
+
3. **Committed Empirical Benchmarks**:
|
|
358
|
+
- The repository tracks empirical performance gains achieved by compiled skills (e.g., documenting GEPA boosting RAG QA accuracy from 80.47% to 100.00% on Ministral 3B).
|
|
359
|
+
|
|
360
|
+
#### C. Modularity & Codebase Structure
|
|
361
|
+
- Highly structured, standard-compliant directory layout:
|
|
362
|
+
```text
|
|
363
|
+
skills/
|
|
364
|
+
├── dspy-fundamentals/
|
|
365
|
+
│ ├── SKILL.md
|
|
366
|
+
│ ├── reference.md
|
|
367
|
+
│ └── example_qa.py
|
|
368
|
+
├── dspy-gepa-optimizer/
|
|
369
|
+
│ ├── SKILL.md
|
|
370
|
+
│ └── example_gepa.py
|
|
371
|
+
tests/
|
|
372
|
+
scripts/
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
#### D. Pros & Cons
|
|
376
|
+
- **Strengths**:
|
|
377
|
+
- *Standardized Agent Packaging*: Full compatibility with Claude Code, Cursor, and Antigravity.
|
|
378
|
+
- *Automated Structural Audits*: Prevents corrupted metadata or broken code examples from reaching agents.
|
|
379
|
+
- *Dry-Run Modes*: Enables CI/CD pipelines to verify code paths without consuming LLM API credits.
|
|
380
|
+
- **Weaknesses & Limitations**:
|
|
381
|
+
- Scope is strictly specialized to the DSPy ecosystem; does not contain browser automation or UI testing primitives.
|
|
382
|
+
|
|
383
|
+
#### E. Token & Compute Efficiency
|
|
384
|
+
- **Token Efficiency**: Maximum. By leveraging progressive disclosure, coding agents only load ~80 tokens of frontmatter metadata during catalog indexing. The full instruction body is loaded into context only when the skill is explicitly activated.
|
|
385
|
+
|
|
386
|
+
#### F. Relevance & Adaptability to Enterprise QA Automation
|
|
387
|
+
- **Score: 9.5 / 10**
|
|
388
|
+
- **Direct QA Mapping**: Serves as the **architectural blueprint for skill packaging in Synapse-Hub**. Synapse-Hub can adopt `dspy-agent-skills`' testing harness (`tests/test_skill_metadata.py` and AST checks) to enforce structural quality gates across its 42 internal skills.
|
|
389
|
+
|
|
390
|
+
---
|
|
391
|
+
|
|
392
|
+
### 2.5 zou-group/textgrad (Automatic Differentiation via Textual Gradients)
|
|
393
|
+
|
|
394
|
+
- **Repository**: `zou-group/textgrad` (`https://github.com/zou-group/textgrad`)
|
|
395
|
+
- **Key Files & Entrypoints**: `textgrad/variable.py`, `textgrad/autograd/llm_ops.py`, `textgrad/autograd/function.py`, `textgrad/optimizer/optimizer.py`, `textgrad/loss.py`.
|
|
396
|
+
- **Publication**: *Nature*, March 2025 (*Optimizing Generative AI Systems with Textual Gradients* by Yuksekgonul et al., Stanford University).
|
|
397
|
+
|
|
398
|
+
```
|
|
399
|
+
+-----------------------------------------------------------------+
|
|
400
|
+
| TextGrad Autograd DAG |
|
|
401
|
+
| |
|
|
402
|
+
| +-----------------------+ |
|
|
403
|
+
| | Variable(Prompt) | |
|
|
404
|
+
| | (requires_grad=True) | |
|
|
405
|
+
| +-----------+-----------+ |
|
|
406
|
+
| | Forward Pass (LLM Completion) |
|
|
407
|
+
| v |
|
|
408
|
+
| +-----------------------+ |
|
|
409
|
+
| | Variable(Generated TC)| |
|
|
410
|
+
| +-----------+-----------+ |
|
|
411
|
+
| | Evaluate Loss |
|
|
412
|
+
| v |
|
|
413
|
+
| +-----------------------+ |
|
|
414
|
+
| | TextLoss(Critique) | (E.g. "Mutant survived at line 42") |
|
|
415
|
+
| +-----------+-----------+ |
|
|
416
|
+
| | |
|
|
417
|
+
| | backward() [Linguistic Chain Rule via LLM] |
|
|
418
|
+
| v |
|
|
419
|
+
| +-----------------------+ |
|
|
420
|
+
| | Textual Gradient | ("Inject boundary assertion for |
|
|
421
|
+
| | (Accumulated on Var)| null payload check") |
|
|
422
|
+
| +-----------+-----------+ |
|
|
423
|
+
| | |
|
|
424
|
+
| | optimizer.step() [TGD Update] |
|
|
425
|
+
| v |
|
|
426
|
+
| +-----------------------+ |
|
|
427
|
+
| | Updated Prompt / Code | |
|
|
428
|
+
| +-----------------------+ |
|
|
429
|
+
+-----------------------------------------------------------------+
|
|
430
|
+
```
|
|
431
|
+
|
|
432
|
+
#### A. Purpose & Core Concept
|
|
433
|
+
TextGrad formulates agent and prompt optimization as a formal analog to **PyTorch Automatic Differentiation**. TextGrad treats strings as computational `Variable` instances forming a directed acyclic graph (DAG). Optimization proceeds by evaluating a natural language loss function and backpropagating textual critiques ("gradients") backward through the DAG using LLMs to implement the chain rule, iteratively updating variables via **Textual Gradient Descent (TGD)**.
|
|
434
|
+
|
|
435
|
+
#### B. Key Architectural Mechanisms
|
|
436
|
+
1. **Computational Graph & `Variable`**:
|
|
437
|
+
- `textgrad/variable.py` defines `Variable(value, role_description, requires_grad=True)`.
|
|
438
|
+
- Variables track their computational lineage via `predecessors` and `grad_fn`. Operations on variables form a DAG recording the forward execution trace.
|
|
439
|
+
2. **Textual Loss Functions (`TextLoss`)**:
|
|
440
|
+
- Evaluates system outputs against domain criteria using an LLM evaluator (e.g., checking code security, edge-case coverage, or adherence to design patterns).
|
|
441
|
+
3. **Backward Pass via Linguistic Chain Rule (`loss.backward()`)**:
|
|
442
|
+
- Traverses the computational graph in reverse topological order.
|
|
443
|
+
- At each operator node (`LLMCall`), an LLM analyzes the downstream textual gradient, the node's input, and the node's output, computing an upstream gradient:
|
|
444
|
+
$$\nabla_{x} \mathcal{L} \approx \text{LLM\_Critique}(\text{upstream\_grad}, x, y)$$
|
|
445
|
+
4. **Textual Gradient Descent (`TGD`)**:
|
|
446
|
+
- `textgrad/optimizer/optimizer.py` aggregates accumulated gradients on a `Variable` and calls an optimizer LLM to revise the variable text:
|
|
447
|
+
$$x^{(t+1)} \leftarrow \text{LLM\_Update}(x^{(t)}, \sum \nabla_x \mathcal{L})$$
|
|
448
|
+
|
|
449
|
+
#### C. Modularity & Codebase Structure
|
|
450
|
+
- `textgrad/autograd/`: Backpropagation engine, custom gradient functions, and operator containers.
|
|
451
|
+
- `textgrad/optimizer/`: Optimizer classes (`TGD`) and prompt templates.
|
|
452
|
+
- `textgrad/engine/`: Multi-backend execution engine supporting OpenAI, Anthropic, Bedrock, and LiteLLM.
|
|
453
|
+
|
|
454
|
+
#### D. Pros & Cons
|
|
455
|
+
- **Strengths**:
|
|
456
|
+
- *PyTorch-Native Mental Model*: Familiar abstractions (`backward()`, `step()`, `zero_grad()`) simplify the design of complex optimization graphs.
|
|
457
|
+
- *Multi-Hop Credit Assignment*: Can assign credit across multi-agent chains, determining whether an error was caused by the planner, the coder, or the tool caller.
|
|
458
|
+
- *Fine-Grained Feedback*: Textual gradients provide explicit semantic directions for improvement rather than coarse scalar rewards.
|
|
459
|
+
- **Weaknesses & Limitations**:
|
|
460
|
+
- *High Token Overhead*: Every backward edge requires an independent LLM call to compute linguistic gradients.
|
|
461
|
+
- *Step-Size Calibration*: Unlike numeric learning rates, textual "learning rates" are difficult to bound, occasionally leading to overshooting or loss of original stylistic tone.
|
|
462
|
+
|
|
463
|
+
#### E. Token & Compute Efficiency
|
|
464
|
+
- **Optimization Phase**: Very high token consumption. A 3-node computational graph with 5 training instances requires $(1 \text{ fwd} + 2 \text{ bwd} + 1 \text{ opt}) \times 5 = 20$ completions per step.
|
|
465
|
+
- **Runtime Inference Phase**: Zero overhead once variables (prompts, Page Classes) are optimized and serialized.
|
|
466
|
+
|
|
467
|
+
#### F. Relevance & Adaptability to Enterprise QA Automation
|
|
468
|
+
- **Score: 8.8 / 10**
|
|
469
|
+
- **Direct QA Mapping**: Synapse-Hub's **Micro-Mutation Testing Engine (`tools/micro_mutator.py`)** generates precise surviving mutant reports. In TextGrad, these reports can act as the loss function $\mathcal{L}_{\text{Mutant}}$, computing textual gradients that backpropagate into test generation prompts to force the synthesis of mutant-killing assertions.
|
|
470
|
+
|
|
471
|
+
---
|
|
472
|
+
|
|
473
|
+
### 2.6 VoltAgent/awesome-agent-skills (Universal Skill Taxonomies & Standards)
|
|
474
|
+
|
|
475
|
+
- **Repository**: `VoltAgent/awesome-agent-skills` (`https://github.com/VoltAgent/awesome-agent-skills`)
|
|
476
|
+
- **Key Files & Entrypoints**: `README.md` (2,011 lines, 243 KB), `CONTRIBUTING.md`.
|
|
477
|
+
- **Maintainers**: VoltAgent & open-source community contributors.
|
|
478
|
+
|
|
479
|
+
#### A. Purpose & Core Concept
|
|
480
|
+
`awesome-agent-skills` is a large-scale, curated ecosystem index cataloging over **1,497 production-grade agent skills** across 40+ engineering teams (including Anthropic, Supabase, Vercel, Sentry, Trail of Bits, and TestMu AI/LambdaTest). It establishes standard quality criteria and path conventions for cross-platform agent skills.
|
|
481
|
+
|
|
482
|
+
#### B. Key Architectural Mechanisms
|
|
483
|
+
1. **Universal Skill Path Standard**:
|
|
484
|
+
- Codifies directory structures across leading agent hosts:
|
|
485
|
+
- Antigravity / Codex: `.agents/skills/` (project) or `~/.gemini/config/skills/` (global).
|
|
486
|
+
- Claude Code: `.claude/skills/`.
|
|
487
|
+
- Cursor: `.cursor/skills/`.
|
|
488
|
+
- Windsurf: `.windsurf/skills/`.
|
|
489
|
+
2. **Quality Gate Guidelines for Agent Skills**:
|
|
490
|
+
- *Third-Person Trigger Descriptions*: Mandates clear trigger keywords to enable accurate routing.
|
|
491
|
+
- *Progressive Disclosure*: Requires frontmatter < 100 tokens and skill body < 500 lines.
|
|
492
|
+
- *No Hardcoded Absolute Paths*: Enforces `$PROJECT_ROOT` or relative path discipline.
|
|
493
|
+
- *Scoped Tool Permissions*: Replaces broad wildcards with explicit tool manifests.
|
|
494
|
+
3. **Specialized QA Domain Taxonomies**:
|
|
495
|
+
- Indexes comprehensive test automation skills authored by TestMu AI (covering Appium, Playwright, REST/GraphQL, Behat, Behave, and CI/CD pipelines).
|
|
496
|
+
|
|
497
|
+
#### C. Modularity & Codebase Structure
|
|
498
|
+
- Pure catalog repository consisting of structured Markdown indexes and links to the hosted web registry index (`https://officialskills.sh/`).
|
|
499
|
+
|
|
500
|
+
#### D. Pros & Cons
|
|
501
|
+
- **Strengths**:
|
|
502
|
+
- Broadest coverage of production skills in the AI ecosystem.
|
|
503
|
+
- High-signal quality criteria that prevent context bloat.
|
|
504
|
+
- Ready-made index of domain-specific QA automation skills.
|
|
505
|
+
- **Weaknesses & Limitations**:
|
|
506
|
+
- Passive catalog without an active runtime, test harness, or automated evaluation pipeline.
|
|
507
|
+
|
|
508
|
+
#### E. Token & Compute Efficiency
|
|
509
|
+
- **Skill Footprint**: Highly efficient; adheres to progressive disclosure.
|
|
510
|
+
- **Compute Overhead**: Zero from the catalog itself.
|
|
511
|
+
|
|
512
|
+
#### F. Relevance & Adaptability to Enterprise QA Automation
|
|
513
|
+
- **Score: 8.5 / 10**
|
|
514
|
+
- **Direct QA Mapping**: Serves as a reference taxonomy for Synapse-Hub's `skills_manifest.json` and informs the directory structure for cross-platform skill exports.
|
|
515
|
+
|
|
516
|
+
---
|
|
517
|
+
|
|
518
|
+
### 2.7 addyosmani/agent-skills (SDLC Lifecycle Disciplines, 3-Tier Evals & Anti-Rationalization)
|
|
519
|
+
|
|
520
|
+
- **Repository**: `addyosmani/agent-skills` (`https://github.com/addyosmani/agent-skills`)
|
|
521
|
+
- **Key Files & Entrypoints**: `skills/`, `commands/`, `agents/`, `references/`, `scripts/run-evals.js`, `scripts/validate-skills.js`, `evals/cases/`.
|
|
522
|
+
- **Author**: Addy Osmani (Google Engineering Lead).
|
|
523
|
+
|
|
524
|
+
```
|
|
525
|
+
+-----------------------------------------------------------------+
|
|
526
|
+
| Addy Osmani 3-Tier Eval Suite |
|
|
527
|
+
| |
|
|
528
|
+
| +---------------------------------------------------------+ |
|
|
529
|
+
| | Tier 1: Schema, Links & Frontmatter Linter (AST Check) | |
|
|
530
|
+
| +----------------------------+----------------------------+ |
|
|
531
|
+
| | |
|
|
532
|
+
| v |
|
|
533
|
+
| +---------------------------------------------------------+ |
|
|
534
|
+
| | Tier 2: Deterministic Routing & Collision Guard (0-Tok) | |
|
|
535
|
+
| | - Light Suffix Stripper & Stopwords Filter | |
|
|
536
|
+
| | - Pairwise Cosine Similarity: Warn >0.50, Fail >=0.75 | |
|
|
537
|
+
| | - Rank-1 Trigger Retrieval Ratchet | |
|
|
538
|
+
| +----------------------------+----------------------------+ |
|
|
539
|
+
| | |
|
|
540
|
+
| v |
|
|
541
|
+
| +---------------------------------------------------------+ |
|
|
542
|
+
| | Tier 3: Behavioral Evals & Headless Execution | |
|
|
543
|
+
| | - Isolated Throwaway Sandbox Sandboxes | |
|
|
544
|
+
| | - Stream-JSON Trace & Rubric Scoring | |
|
|
545
|
+
| +---------------------------------------------------------+ |
|
|
546
|
+
+-----------------------------------------------------------------+
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
#### A. Purpose & Core Concept
|
|
550
|
+
Created by Addy Osmani, this repository codifies Google senior engineering discipline into portable agent skills and slash commands covering the entire software development lifecycle (Define $\rightarrow$ Plan $\rightarrow$ Build $\rightarrow$ Verify $\rightarrow$ Review $\rightarrow$ Ship). It introduces the industry's most rigorous **Three-Tier Evaluation Framework** and **Anti-Rationalization Guards** to prevent agents from taking shortcuts.
|
|
551
|
+
|
|
552
|
+
#### B. Key Architectural Mechanisms
|
|
553
|
+
1. **6-Phase SDLC Lifecycle Mapping**:
|
|
554
|
+
- Modular commands structured across the development lifecycle:
|
|
555
|
+
- `/spec` (`spec-driven-development`, `interview-me`, `constraint-driven-development`).
|
|
556
|
+
- `/plan` (`planning-and-task-breakdown`).
|
|
557
|
+
- `/build` (`test-driven-development`, `context-engineering`, `doubt-driven-development`).
|
|
558
|
+
- `/test` (`browser-testing-with-devtools`, `debugging-and-error-recovery`).
|
|
559
|
+
- `/review` (`code-review-and-quality`, `code-simplification`, `security-and-hardening`).
|
|
560
|
+
- `/ship` (`git-workflow-and-versioning`, `ci-cd-and-automation`).
|
|
561
|
+
2. **Three-Tier Evaluation Framework (`scripts/run-evals.js`)**:
|
|
562
|
+
- **Tier 1 (Schema & Lint)**: Validates YAML frontmatter, link validity, and command argument schema.
|
|
563
|
+
- **Tier 2 (Deterministic Routing & Collision Guard)**:
|
|
564
|
+
- Runs entirely offline without LLM tokens using a lightweight suffix-stripping stemmer heuristic (`scripts/run-evals.js`: *"Light suffix stripping ... Not a real stemmer"*) and TF-IDF vectorizer.
|
|
565
|
+
- Computes pairwise cosine similarity across all skill descriptions. Emits warnings at $>0.50$ and **fails CI builds** at $\ge 0.75$ to prevent trigger ambiguity and skill collision.
|
|
566
|
+
- Tests description retrieval against positive and negative prompt sets, enforcing a Rank-1 routing ratchet.
|
|
567
|
+
- **Tier 3 (Behavioral Evals)**:
|
|
568
|
+
- Opt-in headless agent execution in isolated sandboxes to score behavioral compliance against strict rubrics.
|
|
569
|
+
3. **Anti-Rationalization & Red Flag Anatomy**:
|
|
570
|
+
- Every `SKILL.md` contains an explicit table detailing the excuses agents typically produce when attempting to bypass protocol (e.g., *"The test passes if we ignore the 404"* $\rightarrow$ Rebuttal: *"404 indicates an unhandled routing defect; relax assertion is forbidden"*).
|
|
571
|
+
4. **Isolated Specialist Personas**:
|
|
572
|
+
- Establishes isolated personas (`code-reviewer`, `test-engineer`, `security-auditor`) with a strict architectural invariant: *Personas do not invoke personas*, preventing recursive context explosion.
|
|
573
|
+
|
|
574
|
+
#### C. Modularity & Codebase Structure
|
|
575
|
+
- Exceptionally modular and host-agnostic: core logic lives in `skills/`, with adapters for Claude Code (`.claude/`), Antigravity/Codex (`commands/`), and OpenAI.
|
|
576
|
+
- Validation scripts in `scripts/` are standalone Node.js utilities requiring zero external dependencies.
|
|
577
|
+
|
|
578
|
+
#### D. Pros & Cons
|
|
579
|
+
- **Strengths**:
|
|
580
|
+
- *Zero-Token CI Collision Guard*: Tier-2 eval executes in milliseconds, mathematically preventing skill description overlap without incurring API costs.
|
|
581
|
+
- *Hard Defenses Against AI Laziness*: Anti-rationalization tables directly suppress prompt shortcuts.
|
|
582
|
+
- *End-to-End SDLC Coverage*: Bridges the gap between requirements discovery and release shipping.
|
|
583
|
+
- **Weaknesses & Limitations**:
|
|
584
|
+
- Relies on external test runners or browser tools for physical test dispatch.
|
|
585
|
+
|
|
586
|
+
#### E. Token & Compute Efficiency
|
|
587
|
+
- **Token Efficiency**: Maximum. Tier-2 eval is completely token-free. Skill files are compact (< 300 lines) with checklists segregated into `references/`.
|
|
588
|
+
|
|
589
|
+
#### F. Relevance & Adaptability to Enterprise QA Automation
|
|
590
|
+
- **Score: 9.8 / 10**
|
|
591
|
+
- **Direct QA Mapping**: Addy's **Tier-2 Routing Collision Detection** can be integrated directly into Synapse-Hub's `tools/verify_all.py` (as Quality Gate 10) to ensure that newly authored QA skills never collide with existing skills. Furthermore, the **Anti-Rationalization tables** should be injected into Synapse-Hub subagent prompts (`pom-crafter`, `failure-analyst`) to strictly enforce the Truth Anchor.
|
|
592
|
+
|
|
593
|
+
---
|
|
594
|
+
|
|
595
|
+
### 2.8 mattpocock/skills (Grilling Primitive, Ubiquitous Glossaries & Wayfinder State Machines)
|
|
596
|
+
|
|
597
|
+
- **Repository**: `mattpocock/skills` (`https://github.com/mattpocock/skills`)
|
|
598
|
+
- **Key Files & Entrypoints**: `skills/productivity/grilling/SKILL.md`, `skills/engineering/`, `GLOSSARY.md`, `SCOPE.md`.
|
|
599
|
+
- **Author**: Matt Pocock (Total TypeScript).
|
|
600
|
+
|
|
601
|
+
```
|
|
602
|
+
+-----------------------------------------------------------------+
|
|
603
|
+
| Pocock Grilling Design Tree |
|
|
604
|
+
| |
|
|
605
|
+
| [Root Requirement Spec] |
|
|
606
|
+
| | |
|
|
607
|
+
| +--------------+--------------+ |
|
|
608
|
+
| | | |
|
|
609
|
+
| [Fact vs Decision] [Frontier Calculation] |
|
|
610
|
+
| | | |
|
|
611
|
+
| Is it in the code? Find all prerequisite |
|
|
612
|
+
| Yes -> Agent searches unsettled choices |
|
|
613
|
+
| No -> Ask user | |
|
|
614
|
+
| v |
|
|
615
|
+
| [Batch Question Round] |
|
|
616
|
+
| Format: Question + |
|
|
617
|
+
| Recommended Default |
|
|
618
|
+
| (Typing "yes" accepts) |
|
|
619
|
+
+-----------------------------------------------------------------+
|
|
620
|
+
```
|
|
621
|
+
|
|
622
|
+
#### A. Purpose & Core Concept
|
|
623
|
+
Authored by Matt Pocock, this toolkit provides targeted developer workflow skills designed to counter the two primary failure modes of AI coding assistants: **misalignment** (building the wrong architecture) and **verbosity** (consuming excessive tokens through conversational churn).
|
|
624
|
+
|
|
625
|
+
#### B. Key Architectural Mechanisms
|
|
626
|
+
1. **The Grilling Primitive (`skills/productivity/grilling/SKILL.md`)**:
|
|
627
|
+
- Models requirements gathering as a **design tree**.
|
|
628
|
+
- **Frontier Calculation**: Determines which prerequisite architectural decisions remain open and asks all frontier questions in a single structured round, preventing multi-turn conversational drag.
|
|
629
|
+
- **Recommended Default Formatting**: Formats questions with high-confidence recommended defaults such that the user can accept the entire design proposal simply by typing "yes".
|
|
630
|
+
- **Fact vs. Decision Boundary**: Strictly forbids agents from asking users questions that can be answered by inspecting the codebase. The agent must independently discover facts; only business and strategic decisions are escalated to the user.
|
|
631
|
+
2. **Ubiquitous Language & Domain Modeling (`GLOSSARY.md`)**:
|
|
632
|
+
- Solves prompt verbosity by enforcing a shared, unambiguous project glossary.
|
|
633
|
+
- Replaces long, repetitive natural language explanations with concise domain terms, cutting conversational tokens by 30–50%.
|
|
634
|
+
3. **User-Invoked vs. Model-Invoked Skill Separation**:
|
|
635
|
+
- Distinguishes between user-facing entrypoint skills (`ask-matt`, `grill-me`, `to-spec`) and automated internal discipline skills (`tdd`, `diagnosing-bugs`), preventing recursive tool confusion.
|
|
636
|
+
4. **Wayfinder Multi-Session State Machine**:
|
|
637
|
+
- Decomposes massive codebase migrations into a persistent decision graph on disk, allowing work to resume across context resets without state degradation.
|
|
638
|
+
|
|
639
|
+
#### C. Modularity & Codebase Structure
|
|
640
|
+
- Lightweight, opinionated Markdown skills partitioned into `productivity/`, `engineering/`, and `deprecated/`.
|
|
641
|
+
|
|
642
|
+
#### D. Pros & Cons
|
|
643
|
+
- **Strengths**:
|
|
644
|
+
- *Flawless Requirements Extraction*: The Grilling Primitive eliminates architectural misunderstandings before code is written.
|
|
645
|
+
- *Significant Token Reduction*: Ubiquitous domain glossaries eliminate verbose conversational explanations.
|
|
646
|
+
- *Fact/Decision Invariant*: Stops agents from interrogating the user about obvious codebase facts.
|
|
647
|
+
- **Weaknesses & Limitations**:
|
|
648
|
+
- No automated CI test runner or eval suite included in the repository.
|
|
649
|
+
|
|
650
|
+
#### E. Token & Compute Efficiency
|
|
651
|
+
- **Token Efficiency**: Exceptional. Glossary modeling and frontier question batching minimize token consumption during the specification phase.
|
|
652
|
+
|
|
653
|
+
#### F. Relevance & Adaptability to Enterprise QA Automation
|
|
654
|
+
- **Score: 9.2 / 10**
|
|
655
|
+
- **Direct QA Mapping**: Pocock's **Grilling Primitive** can be embedded directly into Synapse-Hub's `orchestrator` during Stage 1 (Discovery & User Story Extraction) to clarify acceptance criteria and test preconditions before triggering `pom-crafter`.
|
|
656
|
+
|
|
657
|
+
---
|
|
658
|
+
|
|
659
|
+
### 2.9 OpenBMB/ToolBench (Depth-First Search Decision Trees & Tool Retrieval)
|
|
660
|
+
|
|
661
|
+
- **Repository**: `OpenBMB/ToolBench` (`https://github.com/OpenBMB/ToolBench`)
|
|
662
|
+
- **Key Files & Entrypoints**: `toolbench/inference/Algorithms/DFS.py`, `toolbench/inference/qa_pipeline.py`, `toolbench/tooleval/`, `toolbench/retrieval/`, `toolbench/train/`.
|
|
663
|
+
- **Publication**: ICLR 2024 (*ToolBench: An Open Platform for Training, Serving, and Evaluating Large Language Models for Tool Use* by Qin et al., OpenBMB & Tsinghua University).
|
|
664
|
+
|
|
665
|
+
```
|
|
666
|
+
+-----------------------------------------------------------------+
|
|
667
|
+
| ToolBench DFSDT Decision Tree |
|
|
668
|
+
| |
|
|
669
|
+
| [Root Task Node] |
|
|
670
|
+
| | |
|
|
671
|
+
| +--------------+--------------+ |
|
|
672
|
+
| | Action A | Action B |
|
|
673
|
+
| v v |
|
|
674
|
+
| [API Execution] [API Execution] |
|
|
675
|
+
| | | |
|
|
676
|
+
| Error / Failure Observation Received |
|
|
677
|
+
| | | |
|
|
678
|
+
| v v |
|
|
679
|
+
| [Backtrack &] [Evaluate Child Node] |
|
|
680
|
+
| [Prune Branch] | |
|
|
681
|
+
| | v |
|
|
682
|
+
| +--------------------> [Success Path] |
|
|
683
|
+
+-----------------------------------------------------------------+
|
|
684
|
+
```
|
|
685
|
+
|
|
686
|
+
#### A. Purpose & Core Concept
|
|
687
|
+
ToolBench is an open platform for training, serving, and evaluating LLMs on tool-use capabilities across **16,464 real-world RESTful APIs** (from RapidAPI) spanning 49 categories. It addresses the limitation of linear ReAct pipelines by introducing **Depth-First Search Decision Trees (DFSDT)** for algorithmic backtracking during tool execution failures.
|
|
688
|
+
|
|
689
|
+
#### B. Key Architectural Mechanisms
|
|
690
|
+
1. **Depth-First Search Decision Tree (`DFS.py`)**:
|
|
691
|
+
- Replaces fragile linear ReAct loops with a tree-search decision process.
|
|
692
|
+
- When a tool call encounters an error (e.g., HTTP 400, missing parameter, or empty response), DFSDT evaluates branch viability, prunes the failed branch, and **backtracks** to parent nodes to explore alternative tools or parameter combinations.
|
|
693
|
+
- Maintains a node memory tracking tree depth, candidate ranking (`LLM_rank`), and branch pruning state.
|
|
694
|
+
2. **Dense Tool Retrieval (Tool-IR)**:
|
|
695
|
+
- Evaluates a BERT-based dense retrieval model that indexes 16,000+ API documentation schemas.
|
|
696
|
+
- For any user instruction, it dynamically retrieves the top-$K$ relevant API definitions into prompt context, eliminating the need to stuff massive API catalogs into the prompt.
|
|
697
|
+
3. **ToolEval Dual-Metric Evaluation**:
|
|
698
|
+
- Measures two complementary metrics:
|
|
699
|
+
- *Pass Rate*: Proportion of tasks successfully completed within bounded execution steps.
|
|
700
|
+
- *Win Rate*: Pairwise LLM-based preference comparison evaluating trajectory conciseness and execution elegance, showing >80% agreement with human annotators.
|
|
701
|
+
4. **Observation Compression Pipeline**:
|
|
702
|
+
- Implements automated observation truncators and summarizers to prevent massive JSON API payloads from exhausting the context window.
|
|
703
|
+
|
|
704
|
+
#### C. Modularity & Codebase Structure
|
|
705
|
+
- `toolbench/inference/`: Search algorithms (DFS, Single-chain), server abstractions, and execution pipelines.
|
|
706
|
+
- `toolbench/tooleval/`: Automated pass-rate and win-rate evaluators.
|
|
707
|
+
- `toolbench/retrieval/`: API indexing and embedding routines.
|
|
708
|
+
- `toolbench/train/`: Distributed fine-tuning recipes for ToolLLaMA.
|
|
709
|
+
|
|
710
|
+
#### D. Pros & Cons
|
|
711
|
+
- **Strengths**:
|
|
712
|
+
- *Algorithmic Backtracking*: DFSDT allows agents to recover from dead-end tool paths without complete restart.
|
|
713
|
+
- *Massive Scale*: Evaluated across 16,464 APIs and 126k+ instruction instances.
|
|
714
|
+
- *Context Protection*: Dense retrieval and observation truncation protect context budgets.
|
|
715
|
+
- **Weaknesses & Limitations**:
|
|
716
|
+
- *Compute Burden*: Running deep DFSDT searches requires substantial LLM completions.
|
|
717
|
+
- *Dependency on API Stability*: Third-party RapidAPI endpoints can suffer from external network downtime.
|
|
718
|
+
|
|
719
|
+
#### E. Token & Compute Efficiency
|
|
720
|
+
- **Prompt Token Efficiency**: High. Dense retrieval loads only the top-5 API schemas, saving >95% prompt tokens compared to monolithic schema injection.
|
|
721
|
+
- **Search Token Consumption**: Moderate to high when tree backtracking explores multiple alternative branches.
|
|
722
|
+
|
|
723
|
+
#### F. Relevance & Adaptability to Enterprise QA Automation
|
|
724
|
+
- **Score: 8.7 / 10**
|
|
725
|
+
- **Direct QA Mapping**: Synapse-Hub's **Self-Healing Locator (`tools/auto_heal.py`)** can adapt the **DFSDT algorithm** for UI element discovery: when Playwright fails on a primary locator, instead of terminating with a fatal error, the agent backtracks through the Master Locator Hierarchy (ARIA $\rightarrow$ TestId $\rightarrow$ Component Attr $\rightarrow$ CSS $\rightarrow$ XPath).
|
|
726
|
+
|
|
727
|
+
---
|
|
728
|
+
|
|
729
|
+
### 2.10 ShishirPatil/gorilla (BFCL Function Calling Benchmark & GoEx Reversible Execution)
|
|
730
|
+
|
|
731
|
+
- **Repository**: `ShishirPatil/gorilla` (`https://github.com/ShishirPatil/gorilla`)
|
|
732
|
+
- **Key Files & Entrypoints**: `berkeley-function-call-leaderboard/`, `goex/`, `raft/`, `openfunctions/`, `agent-arena/`.
|
|
733
|
+
- **Publication**: UC Berkeley (Patil et al., 2023–2025; *Gorilla: Large Language Model Connected with Massive APIs*).
|
|
734
|
+
|
|
735
|
+
```
|
|
736
|
+
+-----------------------------------------------------------------+
|
|
737
|
+
| Gorilla Ecosystem |
|
|
738
|
+
| |
|
|
739
|
+
| +---------------------------------------------------------+ |
|
|
740
|
+
| | Berkeley Function Calling Leaderboard (BFCL v1 - v4) | |
|
|
741
|
+
| | - AST Argument Validation & Executable Evaluation | |
|
|
742
|
+
| | - Irrelevance Detection & Multi-Turn State Tracking | |
|
|
743
|
+
| +----------------------------+----------------------------+ |
|
|
744
|
+
| | |
|
|
745
|
+
| v |
|
|
746
|
+
| +---------------------------------------------------------+ |
|
|
747
|
+
| | Gorilla Execution Engine (GoEx) & Reversible Sandboxing | |
|
|
748
|
+
| | - Docker Containerized Execution | |
|
|
749
|
+
| | - Damage Confinement (Blocks destructive actions) | |
|
|
750
|
+
| | - Undo Abstraction (Generates reverse actions) | |
|
|
751
|
+
| | - Secret Shielding (Injects credentials post-LLM) | |
|
|
752
|
+
| +---------------------------------------------------------+ |
|
|
753
|
+
+-----------------------------------------------------------------+
|
|
754
|
+
```
|
|
755
|
+
|
|
756
|
+
#### A. Purpose & Core Concept
|
|
757
|
+
Gorilla is the preeminent open ecosystem for connecting LLMs with tools and APIs. It encompasses the **Berkeley Function Calling Leaderboard (BFCL)**—the industry standard for evaluating tool invocation accuracy—and **GoEx (Gorilla Execution Engine)**, a safe runtime environment providing reversible execution and damage confinement for agent actions.
|
|
758
|
+
|
|
759
|
+
#### B. Key Architectural Mechanisms
|
|
760
|
+
1. **Berkeley Function Calling Leaderboard (BFCL v1–v4)**:
|
|
761
|
+
- De facto benchmark evaluating commercial and open-source models across comprehensive functional categories:
|
|
762
|
+
- Simple, multiple, and parallel tool calling.
|
|
763
|
+
- AST structural argument matching.
|
|
764
|
+
- *Irrelevance Detection*: Rigorously tests the model's ability to refrain from invoking tools when the user's prompt does not warrant an API call.
|
|
765
|
+
- Multi-turn stateful dialogues requiring service state retention.
|
|
766
|
+
- Telemetry tracking exact inference cost ($ per 1M tokens) and execution latency.
|
|
767
|
+
2. **Gorilla Execution Engine (GoEx) & Reversible Execution**:
|
|
768
|
+
- `goex/` executes agent actions inside containerized environments with three critical safety primitives:
|
|
769
|
+
- **Undo Abstraction**: For every state-mutating action executed (e.g., database INSERT, file write, or cloud resource provisioning), GoEx automatically records a corresponding reverse action to roll back changes upon task completion.
|
|
770
|
+
- **Damage Confinement**: Analyzes commands before execution to block catastrophic operations (e.g., `rm -rf`, `DROP TABLE`, or unauthorized external network egress).
|
|
771
|
+
- **Secret Shielding**: Strips credentials from LLM context, injecting OAuth2 tokens and private keys directly at the execution proxy layer.
|
|
772
|
+
3. **Retrieval-Augmented Fine-Tuning (RAFT)**:
|
|
773
|
+
- Methodology for teaching models to ignore distractor documentation and generate reasoning paths grounded strictly in verified API reference manuals.
|
|
774
|
+
|
|
775
|
+
#### C. Modularity & Codebase Structure
|
|
776
|
+
- Monorepo design partitioned into independently deployable packages: `berkeley-function-call-leaderboard/` (CLI `bfcl-eval`), `goex/` (sandbox runtime), `raft/` (training pipeline), and `openfunctions/` (model weights).
|
|
777
|
+
|
|
778
|
+
#### D. Pros & Cons
|
|
779
|
+
- **Strengths**:
|
|
780
|
+
- *Gold Standard Function Calling Benchmark*: BFCL provides the most rigorous, cheat-proof tool evaluation available.
|
|
781
|
+
- *Enterprise Safety Primitives*: GoEx's undo abstraction and damage confinement are essential for mission-critical automation.
|
|
782
|
+
- *Irrelevance Verification*: Prevents agents from hallucinating tool invocations.
|
|
783
|
+
- **Weaknesses & Limitations**:
|
|
784
|
+
- Full GoEx execution requires a local Docker engine and certificate authority infrastructure.
|
|
785
|
+
|
|
786
|
+
#### E. Token & Compute Efficiency
|
|
787
|
+
- **Token Efficiency**: High. BFCL profiles cost explicitly. Gorilla OpenFunctions incorporates relevance filters that suppress argument generation when tools are unneeded, saving tokens.
|
|
788
|
+
|
|
789
|
+
#### F. Relevance & Adaptability to Enterprise QA Automation
|
|
790
|
+
- **Score: 9.7 / 10**
|
|
791
|
+
- **Direct QA Mapping**:
|
|
792
|
+
- **GoEx Reversible Teardown**: Maps directly to Synapse-Hub's mandatory QA rule: *"Every creation test case must be paired with automated Teardown cleanup."* GoEx's undo abstraction guarantees zero residual database pollution.
|
|
793
|
+
- **BFCL Benchmark Suite for QA Agents**: Synapse-Hub can adapt BFCL's evaluation framework to benchmark its internal 6 subagents on tool selection accuracy and irrelevance filtering.
|
|
794
|
+
|
|
795
|
+
---
|
|
796
|
+
|
|
797
|
+
## 3. Comprehensive 10-Repository Comparative Matrix
|
|
798
|
+
|
|
799
|
+
### 3.1 Multi-Dimensional Comparative Evaluation Table
|
|
800
|
+
|
|
801
|
+
The following matrix compares all 10 surveyed repositories across 10 rigorous architectural and operational dimensions:
|
|
802
|
+
|
|
803
|
+
| # | Repository | Core Paradigm | Learning & Optimization Signal | Optimization Target | Architectural Modularity | Training Token Efficiency | Runtime Token Efficiency | Error Recovery & Self-Healing | Enterprise QA Relevance |
|
|
804
|
+
|:---:|:---|:---|:---|:---|:---|:---|:---|:---|:---:|
|
|
805
|
+
| **1** | **MineDojo/Voyager** | Lifelong Embodied Learning & Skill Library | Environment observation diffs + Compiler errors | Executable JS code programs (in ChromaDB) | 4 decoupled agents (Action, Critic, Curric, Skill) | **Low** (8k–25k tokens per task exploration) | **Moderate** (Top-5 vector skill injection) | Iterative code repair with execution feedback | **9.2 / 10** (Clean POM & Live Explorer) |
|
|
806
|
+
| **2** | **noahshinn/Reflexion** | Verbal Reinforcement Learning (No Weights) | Unit test stacktraces + Natural language reflections | Short-term episodic memory & code trajectories | Decoupled Actor-Evaluator-Reflection triad | **High** (1 reflection + 1 retry per failure) | **High** (Bounded episodic memory buffer) | Verbal reflection diagnosing failure root causes | **9.8 / 10** (Failure Autopsy & Auto-Heal) |
|
|
807
|
+
| **3** | **stanfordnlp/dspy** | Declarative LM Programming & Compilation | Objective scalar evaluation metrics (DoD Gates) | System prompts, few-shot exemplars, LM weights | Composable PyTorch-style modules & optimizers | **Low / Heavy** (50k–300k tokens during compile) | **Maximum** (Frozen compiled prompt artifact) | Computational assertions with auto-backtracking | **9.9 / 10** (Quality Gate DoD Compilation) |
|
|
808
|
+
| **4** | **intertwine/dspy-agent-skills** | Spec-Compliant Agent Skill Packaging | Static frontmatter linters + GEPA prompt evolution | Modular agent skill directory structures | Standardized `SKILL.md` + references + examples | **Low** (During offline GEPA compile runs) | **Maximum** (Progressive disclosure <100 tok) | Frontmatter schema & AST validation harness | **9.5 / 10** (Skill Standard & Manifests) |
|
|
809
|
+
| **5** | **zou-group/textgrad** | Automatic Differentiation via Textual Gradients | Natural language loss functions via LLM critique | Arbitrary string variables (prompts, code, answers) | PyTorch-style Autograd DAG (`Variable`, `backward`) | **Very Low** (LLM call per backward DAG edge) | **Maximum** (Zero overhead for optimized strings) | Textual gradient descent parameter updates | **8.8 / 10** (Mutant-Directed Assertions) |
|
|
810
|
+
| **6** | **VoltAgent/awesome-agent-skills** | Curated Universal Skill Taxonomy | Community peer review & schema guidelines | Standardized Markdown skill repositories | Decentralized index linking external repositories | **N/A** (Pure catalog metadata) | **Maximum** (Adheres to progressive disclosure) | N/A (Manual curation) | **8.5 / 10** (Skill Taxonomy & Discovery) |
|
|
811
|
+
| **7** | **addyosmani/agent-skills** | SDLC Discipline, 3-Tier Evals & Anti-Rationalization | Tier-2 Cosine similarity (0-tok) + Headless evals | Portable agent skills & multi-host commands | Decoupled skills, commands, references, scripts | **Maximum** (Tier-2 runs offline in ms with 0 tokens) | **Maximum** (Concise skills + progressive disclosure) | Anti-rationalization tables & floor guards | **9.8 / 10** (DoD Gates & Collision Guard) |
|
|
812
|
+
| **8** | **mattpocock/skills** | Senior Dev Workflow & Grilling Primitive | Design-tree prerequisite frontier calculation | Project domain models & requirements specs | Lifecycle-partitioned Markdown files | **Maximum** (Question batching slashes roundtrips) | **Maximum** (Glossary eliminates conversational bloat) | Systematic hypothesis-gated debugging | **9.2 / 10** (Requirements Discovery & Glossaries) |
|
|
813
|
+
| **9** | **OpenBMB/ToolBench** | Decision Tree Search & Dense Tool Retrieval | Pass Rate & Win Rate preference metrics | Tool execution trajectories & ToolLLaMA weights | Decoupled inference, retrieval, and eval modules | **Low** (Requires multi-GPU A100 training) | **High** (Dense tool retrieval + observation truncate) | DFSDT algorithmic backtracking on failure | **8.7 / 10** (Backtracking Locator Healing) |
|
|
814
|
+
| **10** | **ShishirPatil/gorilla** | Function Calling Benchmark & Reversible Sandboxing | AST argument matching, execution diffs & cost | Tool-calling model weights & GoEx execution traces | Modular monorepo (`bfcl-eval`, `goex`, `raft`) | **Moderate** (Evaluation runs) to **Low** (Training) | **High** (Relevance filters suppress bad args) | GoEx undo rollback & damage confinement | **9.7 / 10** (Reversible Teardown & Benchmarks) |
|
|
815
|
+
|
|
816
|
+
### 3.2 Quantitative Capability & QA Adaptability Scoring
|
|
817
|
+
|
|
818
|
+
To establish clear integration priorities, the 10 repositories are scored across 5 key enterprise automation dimensions (each rated from 1 to 10):
|
|
819
|
+
- **QA Relevance**: Alignment with enterprise test automation workflows (POM, assertions, locators, runners).
|
|
820
|
+
- **Execution Safety**: Robustness against unintended destructive actions, data pollution, and security leaks.
|
|
821
|
+
- **Token Efficiency**: Economy of token consumption across training/compilation and runtime inference.
|
|
822
|
+
- **Architectural Rigor**: Modularity, formal abstractions, testability, and deterministic behavior.
|
|
823
|
+
- **Implementation Feasibility**: Ease of integration into Synapse-Hub without disruptive architectural rewrites.
|
|
824
|
+
|
|
825
|
+
```
|
|
826
|
+
Repository Score Breakdown (Equal-Weighted Average, w_i = 0.20 / 10.0):
|
|
827
|
+
1. stanfordnlp/dspy : [QA: 10, Safety: 9, Token: 9, Rigor: 10, Feas: 9] -> 9.4 / 10
|
|
828
|
+
2. addyosmani/agent-skills : [QA: 10, Safety: 10, Token: 10, Rigor: 9, Feas: 10] -> 9.8 / 10
|
|
829
|
+
3. noahshinn/Reflexion : [QA: 10, Safety: 9, Token: 9, Rigor: 9, Feas: 10] -> 9.4 / 10
|
|
830
|
+
4. ShishirPatil/gorilla : [QA: 10, Safety: 10, Token: 8, Rigor: 10, Feas: 9] -> 9.4 / 10
|
|
831
|
+
5. intertwine/dspy-agent-sk : [QA: 9, Safety: 9, Token: 10, Rigor: 10, Feas: 9] -> 9.4 / 10
|
|
832
|
+
6. MineDojo/Voyager : [QA: 9, Safety: 8, Token: 7, Rigor: 9, Feas: 8] -> 8.2 / 10
|
|
833
|
+
7. mattpocock/skills : [QA: 9, Safety: 9, Token: 10, Rigor: 8, Feas: 10] -> 9.2 / 10
|
|
834
|
+
8. OpenBMB/ToolBench : [QA: 9, Safety: 8, Token: 8, Rigor: 9, Feas: 8] -> 8.4 / 10
|
|
835
|
+
9. zou-group/textgrad : [QA: 9, Safety: 8, Token: 6, Rigor: 9, Feas: 7] -> 7.8 / 10
|
|
836
|
+
10. VoltAgent/awesome-agent : [QA: 8, Safety: 8, Token: 10, Rigor: 7, Feas: 10] -> 8.6 / 10
|
|
837
|
+
```
|
|
838
|
+
|
|
839
|
+
---
|
|
840
|
+
|
|
841
|
+
## 4. Cross-Cutting Analysis of 5 Core Training & Optimization Paradigms
|
|
842
|
+
|
|
843
|
+
### 4.1 Paradigm 1: Curriculum Learning & Lifelong Skill Libraries
|
|
844
|
+
|
|
845
|
+
- **Primary Representatives**: MineDojo/Voyager, OpenBMB/ToolBench.
|
|
846
|
+
- **Core Mechanism**: Rather than relying on static prompt instructions, the agent autonomously navigates a curriculum of increasing difficulty. Verified executable programs are indexed into vector or relational databases and dynamically retrieved to compose solutions for complex, unfamiliar objectives.
|
|
847
|
+
- **Enterprise QA Application**:
|
|
848
|
+
- In enterprise web automation, applications exhibit strict state dependencies: testing checkout requires an authenticated session with master data and inventory seeded.
|
|
849
|
+
- A QA Curriculum Engine automatically stages test scenarios: starting with boundary checks on simple leaf inputs, progressing to CRUD operations, and culminating in multi-role approval workflows.
|
|
850
|
+
- Newly synthesized and verified Page Class methods are persisted into a **Lifelong SkillVault**, eliminating redundant locator discovery.
|
|
851
|
+
|
|
852
|
+
### 4.2 Paradigm 2: Verbal Reinforcement Learning & Multi-Trial Reflection Loops
|
|
853
|
+
|
|
854
|
+
- **Primary Representatives**: noahshinn/Reflexion, stanfordnlp/dspy (GEPA).
|
|
855
|
+
- **Core Mechanism**: Eliminates mathematical gradient descent by converting environmental failures (compiler errors, JUnit stacktraces, DOM timeouts) into natural language critiques stored in episodic memory buffers. In subsequent trials, the agent conditions its generation on these reflections, systematically avoiding identical mistakes.
|
|
856
|
+
- **Enterprise QA Application**:
|
|
857
|
+
- Direct integration into **Closed-Loop Failure Healing (`auto_heal.py`)**.
|
|
858
|
+
- When Playwright tests fail in headless CI runs, the failure analyzer extracts the JUnit XML and visual screenshot diff, synthesizes a concise 2-line reflection, and executes a multi-trial healing loop ($K=3$) before committing patches to the shared SQLite memory cache.
|
|
859
|
+
|
|
860
|
+
### 4.3 Paradigm 3: Declarative LM Programming & Teleprompter Prompt Compilation
|
|
861
|
+
|
|
862
|
+
- **Primary Representatives**: stanfordnlp/dspy, intertwine/dspy-agent-skills.
|
|
863
|
+
- **Core Mechanism**: Replaces fragile, hand-tuned prompt strings with declarative signatures separating input/output schema from underlying execution. Teleprompters compile prompt pipelines by searching instruction and few-shot demonstration space against objective evaluation metrics.
|
|
864
|
+
- **Enterprise QA Application**:
|
|
865
|
+
- Synapse-Hub's **9 Quality Gates DoD (`tools/verify_all.py`)** and **Micro-Mutator Score ($\ge 80\%$)** provide the exact deterministic scalar reward required for prompt compilation.
|
|
866
|
+
- Prompt compilation runs offline during release builds, freezing high-performance instructions that run with zero optimization tokens in production.
|
|
867
|
+
|
|
868
|
+
### 4.4 Paradigm 4: Automatic Differentiation & Textual Gradients
|
|
869
|
+
|
|
870
|
+
- **Primary Representatives**: zou-group/textgrad.
|
|
871
|
+
- **Core Mechanism**: Formulates prompt optimization as backpropagation over a computational graph. Natural language loss functions compute linguistic gradients that backpropagate through intermediate agent nodes to iteratively update prompts or code variables via Textual Gradient Descent.
|
|
872
|
+
- **Enterprise QA Application**:
|
|
873
|
+
- Mutation testing (`micro_mutator.py`) provides explicit textual feedback identifying surviving mutants (e.g., *"Boundary mutant on line 88 survived: test checked status code 200 but failed to assert returned JSON payload"*).
|
|
874
|
+
- This critique acts as a textual gradient vector $\nabla_{\text{prompt}} \mathcal{L}_{\text{QA}}$ that directly refines test generation prompts to enforce sharp, mutant-killing assertions.
|
|
875
|
+
|
|
876
|
+
### 4.5 Paradigm 5: Large-Scale Tool-Use, Function Calling & Sandboxed Execution
|
|
877
|
+
|
|
878
|
+
- **Primary Representatives**: ShishirPatil/gorilla, OpenBMB/ToolBench.
|
|
879
|
+
- **Core Mechanism**: Rigorous benchmarking of function calling across multi-turn stateful environments, coupled with containerized sandboxes providing damage confinement, irrelevance detection, and automatic undo/rollback of stateful side-effects.
|
|
880
|
+
- **Enterprise QA Application**:
|
|
881
|
+
- Enforces Synapse-Hub's non-negotiable invariant: *"Every creation test case must execute automated Teardown cleanup."*
|
|
882
|
+
- Gorilla GoEx's **undo abstraction** records database inserts, entity creations, and filesystem modifications during test runs, automatically generating rollback operations to guarantee zero database pollution.
|
|
883
|
+
|
|
884
|
+
---
|
|
885
|
+
|
|
886
|
+
## 5. Synapse-Hub Current Architecture Baseline
|
|
887
|
+
|
|
888
|
+
Synapse-Hub (version 2.5.0) provides an enterprise-grade automated QA framework designed around strict architectural invariants, truth anchors, and context engineering.
|
|
889
|
+
|
|
890
|
+
```
|
|
891
|
+
+-----------------------------------------------------------------+
|
|
892
|
+
| Synapse-Hub Architecture |
|
|
893
|
+
| |
|
|
894
|
+
| +---------------------------------------------------------+ |
|
|
895
|
+
| | Decoupled Hub-and-Spoke Model | |
|
|
896
|
+
| | - Hub: Universal Skills, Swarm Agents, 90 Tools, MCP | |
|
|
897
|
+
| | - Spoke: Target Project Repo (Clean POM, Zero Agents) | |
|
|
898
|
+
| +----------------------------+----------------------------+ |
|
|
899
|
+
| | |
|
|
900
|
+
| v |
|
|
901
|
+
| +---------------------------------------------------------+ |
|
|
902
|
+
| | 5-Stage Deterministic FSM Pipeline | |
|
|
903
|
+
| | Stage 1: Discovery (live-explorer, Playwright Live) | |
|
|
904
|
+
| | Stage 1.5: Spec Review Gate (goldfish_review.py) | |
|
|
905
|
+
| | Stage 2: Code Synthesis (pom-crafter, Clean POM, IR) | |
|
|
906
|
+
| | Stage 3: Headless Execution (test-runner, CLI runners) | |
|
|
907
|
+
| | Stage 4: Failure Autopsy & Healing (failure-analyst) | |
|
|
908
|
+
| | Stage 5: Final Acceptance (quality-gatekeeper, 9 Gates)| |
|
|
909
|
+
| +----------------------------+----------------------------+ |
|
|
910
|
+
| | |
|
|
911
|
+
| v |
|
|
912
|
+
| +---------------------------------------------------------+ |
|
|
913
|
+
| | Deterministic Verification & Episodic Memory | |
|
|
914
|
+
| | - 9 Quality Gates DoD (tools/verify_all.py) | |
|
|
915
|
+
| | - AST Linters (infer_linter.py, micro_mutator.py) | |
|
|
916
|
+
| | - Dual SQLite Memory Cache (cache_resolver.py) | |
|
|
917
|
+
| +---------------------------------------------------------+ |
|
|
918
|
+
+-----------------------------------------------------------------+
|
|
919
|
+
```
|
|
920
|
+
|
|
921
|
+
### 5.1 Hub-and-Spoke Decoupled Operating Model
|
|
922
|
+
- **Centralized Hub (`agent-skills`)**: An independent, standalone repository containing 42 standardized agent skills, 6 specialized subagent profiles, 90 CLI tools, and an MCP server.
|
|
923
|
+
- **Decoupled Spokes (Target Projects)**: Target testing repositories mount Synapse-Hub as an external workspace or CLI dependency. Target project Git trees contain exclusively target test code (Page Classes, specs, fixtures) with zero `.agents/` metadata pollution.
|
|
924
|
+
|
|
925
|
+
### 5.2 Clean POM, Flat Architecture & Universal IR
|
|
926
|
+
- **Clean Page Object Model (POM)**: Test scripts contain strictly business workflow method calls on Page Classes. Raw XPaths, CSS selectors, and framework-specific finders are strictly forbidden in test scripts.
|
|
927
|
+
- **Flat Architecture**: All test cases reside directly in parent directories (`TC-<Number>_<Name>`), eliminating nested directory complexity.
|
|
928
|
+
- **Master Locator Hierarchy (6 Tiers)**:
|
|
929
|
+
$$\text{Tier 1: ARIA Semantic} \rightarrow \text{Tier 2: TestId} \rightarrow \text{Tier 3: ID/Name} \rightarrow \text{Tier 4: Comp Attr} \rightarrow \text{Tier 5: Resilient CSS} \rightarrow \text{Tier 6: Rel XPath}$$
|
|
930
|
+
- **Universal IR (`tools/universal_ir.py`)**: Framework-agnostic JSON Intermediate Representation decoupling test intent from target languages (Playwright TS/Python, Katalon Groovy, Cypress).
|
|
931
|
+
|
|
932
|
+
### 5.3 Playwright Live Interaction & Fast-Loop Discovery
|
|
933
|
+
- **Three-Stage Lifecycle**:
|
|
934
|
+
1. *Discovery*: Probe accessibility trees (`accessibility.snapshot()`) and compact AOM (< 500 tokens) in pure headless mode.
|
|
935
|
+
2. *Script Creation*: Require 100% selector hit rate under 200ms on live DOM before codifying into Page Classes.
|
|
936
|
+
3. *In-Place Debugging*: Keep browser paused at failure points to diagnose overlays and DOM detachments in-place, eliminating slow re-runs.
|
|
937
|
+
|
|
938
|
+
### 5.4 Verification Discipline & The 9 Quality Gates DoD
|
|
939
|
+
Enforced by `tools/verify_all.py`:
|
|
940
|
+
- `Gate 0`: Author Attribution & MIT License Integrity (`author_integrity.py`).
|
|
941
|
+
- `Gate 1`: 100% UTF-8 No BOM across all source and test files.
|
|
942
|
+
- `Gate 2`: Path Portability (blocks hardcoded absolute machine paths).
|
|
943
|
+
- `Gate 3`: Clean POM Flat Architecture (blocks raw selectors in test scripts).
|
|
944
|
+
- `Gate 4`: AI Docs Anti-Drift (reconciles AST classes against `PROJECT_AI_GUIDE.md`).
|
|
945
|
+
- `Gate 5`: Temporary File Blacklist (prevents build junk in git index).
|
|
946
|
+
- `Gate 6`: Credential Security (scans for hardcoded secrets).
|
|
947
|
+
- `Gate 7`: Static Defect & Fake-Pass Sentinel (`tools/infer_linter.py` AST scan blocking 0-assertion tests, empty catch blocks, and hard sleeps).
|
|
948
|
+
- `Gate 8`: Assertion Sensitivity Sentinel (`tools/micro_mutator.py` AST mutation testing enforcing Mutation Score $\ge 80\%$).
|
|
949
|
+
|
|
950
|
+
### 5.5 Dual-Storage SQLite Episodic & Semantic Memory Cache
|
|
951
|
+
- **Dual SQLite Database Architecture**:
|
|
952
|
+
- Local database (`.synapse/cache.db`): Scoped to target project, protecting corporate NDA/IP.
|
|
953
|
+
- Global database (`~/.agent_memory/global_cache.db`): Cross-session shared memory.
|
|
954
|
+
- **Empirical Memory State**: Contains 7 verified positive patches, 1,585 blacklisted negative fixes, and 45 episodic lessons learned.
|
|
955
|
+
- **Error Fingerprinting**: Generates deterministic DJB2 hash fingerprints (`fp_<hash>`) after sanitizing line numbers, timestamps, and memory hex addresses.
|
|
956
|
+
|
|
957
|
+
### 5.6 6-Agent Swarm System & Single Front-Door FSM
|
|
958
|
+
Incoming tasks enter exclusively through the **Single Front-Door (`orchestrator`)** in a strictly sequential 5-stage pipeline:
|
|
959
|
+
$$\text{Stage 1: Discovery (live-explorer)} \rightarrow \text{Stage 1.5: Goldfish Spec Review Gate} \rightarrow \text{Stage 2: Code Synthesis (pom-crafter)} \rightarrow \text{Stage 3: Headless Execution (test-runner)} \rightarrow \text{Stage 4: Failure Autopsy & Healing (failure-analyst)} \rightarrow \text{Stage 5: Final Acceptance DoD (quality-gatekeeper)}$$
|
|
960
|
+
|
|
961
|
+
### 5.7 The 6 Pillars of Context Engineering & Token Optimization
|
|
962
|
+
1. **Observation Masking**: Truncates 95% of CLI logs, replacing them with disk pointers.
|
|
963
|
+
2. **Progressive Disclosure**: Loads context in 3 tiers (Repo map <150 lines $\rightarrow$ 1-hop AST graph $\rightarrow$ 20–50 line micro-reads).
|
|
964
|
+
3. **KV-Cache Friendly Ordering**: Immutable static prefixes at the top; dynamic state appended at the bottom.
|
|
965
|
+
4. **Scoped Context & Compaction Anchoring**: Persistent disk state in `PROJECT_STATE.md` survives context compaction.
|
|
966
|
+
5. **Semantic Context Distillation**: Compact AOM extraction (< 500 tokens) via `tools/browser_helpers.js`.
|
|
967
|
+
6. **Elephant-Goldfish Memory Decoupling**: Separates stateful deep-context planning (The Elephant) from stateless, zero-context adversarial specification review (The Goldfish via `tools/goldfish_review.py`).
|
|
968
|
+
|
|
969
|
+
---
|
|
970
|
+
|
|
971
|
+
## 6. Architectural Integration Blueprint for Synapse-Hub
|
|
972
|
+
|
|
973
|
+
### 6.1 Target Architecture Blueprint & System Topology
|
|
974
|
+
|
|
975
|
+
The blueprint integrates the most valuable mechanisms from the 10 surveyed repositories into Synapse-Hub's existing architecture, hardening every stage of the 5-Stage FSM:
|
|
976
|
+
|
|
977
|
+
```mermaid
|
|
978
|
+
flowchart TD
|
|
979
|
+
User([User / System Task]) --> Orchestrator["<b>Lead QA Orchestrator</b><br/><i>Single Front-Door Entrypoint</i><br/>+ Pocock Grilling Primitive (Frontier Calc)"]
|
|
980
|
+
|
|
981
|
+
subgraph FSM["Enhanced 5-Stage Gatekeeper Pipeline"]
|
|
982
|
+
Stage1["<b>Stage 1: Discovery</b><br/>• Subagent: live-explorer<br/>• Playwright Live AOM (<200ms)<br/>• SkillVault Semantic Retrieval (Voyager)"]
|
|
983
|
+
|
|
984
|
+
Stage15{"<b>Stage 1.5: Spec Review Gate</b><br/>• Goldfish Review (Elephant-Goldfish)<br/>• Ubiquitous Glossary Check (Pocock)"}
|
|
985
|
+
|
|
986
|
+
Stage2["<b>Stage 2: Code Synthesis</b><br/>• Subagent: pom-crafter<br/>• Clean POM + Universal IR<br/>• DSPy Compiled Signatures (MIPROv2)<br/>• Anti-Rationalization Guard (Addy)"]
|
|
987
|
+
|
|
988
|
+
Stage3["<b>Stage 3: Headless Execution</b><br/>• Subagent: test-runner<br/>• Reversible Execution Sandbox (GoEx Undo)<br/>• DFSDT Backtracking Runner (ToolBench)<br/>• Distill Verbose CLI Logs"]
|
|
989
|
+
|
|
990
|
+
Stage4["<b>Stage 4: Failure Autopsy & Healing</b><br/>• Subagent: failure-analyst<br/>• Multi-Trial Episodic Reflexion Loop (K=3)<br/>• Surviving Mutant TextGrad Feedback<br/>• Cache Resolver SQLite Memory Update"]
|
|
991
|
+
|
|
992
|
+
Stage5["<b>Stage 5: Final Acceptance (DoD)</b><br/>• Subagent: quality-gatekeeper<br/>• 10 Quality Gates DoD<br/>• Gate 10: Tier-2 Skill Collision Guard (Addy)<br/>• Mutation Score >= 80% (micro_mutator)"]
|
|
993
|
+
|
|
994
|
+
Stage1 --> Stage15
|
|
995
|
+
Stage15 -->|Approved| Stage2
|
|
996
|
+
Stage15 -->|Rejected| Stage1
|
|
997
|
+
Stage2 --> Stage3
|
|
998
|
+
Stage3 -->|Pass| Stage5
|
|
999
|
+
Stage3 -->|Fail| Stage4
|
|
1000
|
+
Stage4 -->|Healed Trajectory| Stage3
|
|
1001
|
+
Stage4 -->|Exhausted K=3| FailureReport([AUT Defect Ticket Created])
|
|
1002
|
+
Stage5 -->|All 10 Gates Pass| VerifiedCommit([Verified Clean Git Commit])
|
|
1003
|
+
end
|
|
1004
|
+
|
|
1005
|
+
Orchestrator --> Stage1
|
|
1006
|
+
```
|
|
1007
|
+
|
|
1008
|
+
### 6.2 Subsystem 1: SkillVault Lifelong POM Library (Voyager Pattern)
|
|
1009
|
+
|
|
1010
|
+
#### A. Architectural Design
|
|
1011
|
+
Adapts Voyager's `SkillManager` into a persistent SQLite/Chroma store (`.synapse/skill_vault.db`) that stores verified Page Class methods and business interaction workflows. When `pom-crafter` designs a new test case, it queries `SkillVault` using semantic embeddings or BM25 keyword matching to retrieve existing verified methods, preventing duplicate locator authoring.
|
|
1012
|
+
|
|
1013
|
+
#### B. Component Interface & Data Contract
|
|
1014
|
+
```python
|
|
1015
|
+
# tools/skill_vault.py
|
|
1016
|
+
from dataclasses import dataclass
|
|
1017
|
+
from typing import List, Optional
|
|
1018
|
+
import sqlite3
|
|
1019
|
+
|
|
1020
|
+
@dataclass
|
|
1021
|
+
class PageClassPrimitive:
|
|
1022
|
+
skill_id: str # e.g., "SKILL_AUTH_LOGIN_SSO"
|
|
1023
|
+
page_class: str # e.g., "LoginPage"
|
|
1024
|
+
method_signature: str # e.g., "loginWithSSO(username: str, sso_token: str)"
|
|
1025
|
+
method_code: str # Complete executable method body
|
|
1026
|
+
locator_tier: int # 1 to 6 (Master Locator Hierarchy)
|
|
1027
|
+
semantic_docstring: str # Generated docstring for embedding
|
|
1028
|
+
verified_on_url: str # Target AUT URL verified against
|
|
1029
|
+
mutation_score: float # Verified assertion sensitivity score
|
|
1030
|
+
|
|
1031
|
+
class SkillVault:
|
|
1032
|
+
def __init__(self, db_path: str = ".synapse/skill_vault.db"):
|
|
1033
|
+
self.conn = sqlite3.connect(db_path)
|
|
1034
|
+
self._init_schema()
|
|
1035
|
+
|
|
1036
|
+
def query_similar_primitives(self, query: str, top_k: int = 5) -> List[PageClassPrimitive]:
|
|
1037
|
+
"""Performs BM25 / Vector search over verified Page Class methods."""
|
|
1038
|
+
...
|
|
1039
|
+
|
|
1040
|
+
def register_verified_primitive(self, primitive: PageClassPrimitive) -> bool:
|
|
1041
|
+
"""Persists a new primitive after passing Gate 7 (infer_linter) and Gate 8 (micro_mutator)."""
|
|
1042
|
+
...
|
|
1043
|
+
```
|
|
1044
|
+
|
|
1045
|
+
### 6.3 Subsystem 2: Multi-Trial Episodic Reflexion Loop in `auto_heal.py` (Reflexion Pattern)
|
|
1046
|
+
|
|
1047
|
+
#### A. Architectural Design
|
|
1048
|
+
Replaces single-shot heuristic locator healing in `tools/auto_heal.py` with an iterative, multi-trial reflection loop ($K=3$). When a test fails in Stage 3:
|
|
1049
|
+
1. `Evaluator`: Extracts JUnit XML stacktrace and failure screenshot.
|
|
1050
|
+
2. `Reflexion Generator`: Prompts an LLM to generate a structured 2-line verbal diagnosis of why the selector or assertion failed.
|
|
1051
|
+
3. `Episodic Buffer`: Accumulates reflections into memory.
|
|
1052
|
+
4. `Actor`: Re-generates the locator or method patch conditioned on accumulated reflections.
|
|
1053
|
+
5. `Verification`: If verified, patch is written to positive memory (`healing_cache`); if all 3 trials fail, patch is added to negative memory (`failed_fixes`).
|
|
1054
|
+
|
|
1055
|
+
#### B. Component Interface & Data Contract
|
|
1056
|
+
```python
|
|
1057
|
+
# tools/auto_heal_reflexion.py
|
|
1058
|
+
from dataclasses import dataclass
|
|
1059
|
+
from typing import List, Optional
|
|
1060
|
+
|
|
1061
|
+
@dataclass
|
|
1062
|
+
class ReflexionTrial:
|
|
1063
|
+
trial_number: int
|
|
1064
|
+
patch_attempt: str
|
|
1065
|
+
error_feedback: str
|
|
1066
|
+
verbal_reflection: str
|
|
1067
|
+
is_passing: bool
|
|
1068
|
+
|
|
1069
|
+
class ClosedLoopHealer:
|
|
1070
|
+
def __init__(self, max_trials: int = 3):
|
|
1071
|
+
self.max_trials = max_trials
|
|
1072
|
+
self.memory_buffer: List[ReflexionTrial] = []
|
|
1073
|
+
|
|
1074
|
+
def execute_healing_loop(self, failing_test: str, initial_stacktrace: str) -> bool:
|
|
1075
|
+
candidate_patch: str = ""
|
|
1076
|
+
for t in range(1, self.max_trials + 1):
|
|
1077
|
+
reflection = self._generate_reflection(initial_stacktrace, self.memory_buffer)
|
|
1078
|
+
candidate_patch = self._synthesize_patch(failing_test, reflection, self.memory_buffer)
|
|
1079
|
+
is_pass, new_error = self._verify_patch_headless(failing_test, candidate_patch)
|
|
1080
|
+
|
|
1081
|
+
trial = ReflexionTrial(t, candidate_patch, new_error, reflection, is_pass)
|
|
1082
|
+
self.memory_buffer.append(trial)
|
|
1083
|
+
|
|
1084
|
+
if is_pass:
|
|
1085
|
+
self._record_positive_memory(candidate_patch, reflection)
|
|
1086
|
+
return True
|
|
1087
|
+
|
|
1088
|
+
self._record_negative_memory(candidate_patch, f"Failed after {self.max_trials} reflexion trials")
|
|
1089
|
+
return False
|
|
1090
|
+
```
|
|
1091
|
+
|
|
1092
|
+
### 6.4 Subsystem 3: Declarative QA Signature Compilation Engine (DSPy Pattern)
|
|
1093
|
+
|
|
1094
|
+
#### A. Architectural Design
|
|
1095
|
+
Replaces brittle static prompt files for `pom-crafter` with declarative DSPy signatures. Uses `BootstrapFewShot` and `MIPROv2` teleprompters to compile system prompts and few-shot exemplars, using the **9 Quality Gates DoD pass rate** as the quantitative objective function.
|
|
1096
|
+
|
|
1097
|
+
#### B. Component Interface & Declarative Signatures
|
|
1098
|
+
```python
|
|
1099
|
+
# tools/dspy_qa_signatures.py
|
|
1100
|
+
import dspy
|
|
1101
|
+
|
|
1102
|
+
class UserStoryToUniversalIR(dspy.Signature):
|
|
1103
|
+
"""Compile human acceptance criteria into Universal QA Testing IR."""
|
|
1104
|
+
user_story: str = dspy.InputField(desc="User story with acceptance criteria")
|
|
1105
|
+
aom_snapshot: str = dspy.InputField(desc="Live AOM tree of target page")
|
|
1106
|
+
universal_ir_json: str = dspy.OutputField(desc="Valid Universal IR JSON matching tools/universal_ir.py")
|
|
1107
|
+
|
|
1108
|
+
class UniversalIRToCleanPOM(dspy.Signature):
|
|
1109
|
+
"""Transpile Universal IR into multi-language Page Classes adhering to Flat Architecture."""
|
|
1110
|
+
universal_ir_json: str = dspy.InputField(desc="Validated Universal IR JSON")
|
|
1111
|
+
target_framework: str = dspy.InputField(desc="Playwright_TS, Playwright_Py, or Katalon_Groovy")
|
|
1112
|
+
page_class_code: str = dspy.OutputField(desc="Executable BasePage subclass with zero raw XPaths")
|
|
1113
|
+
test_script_code: str = dspy.OutputField(desc="Clean test script containing strictly Page Class calls")
|
|
1114
|
+
|
|
1115
|
+
def qa_dod_metric(gold, pred, trace=None) -> float:
|
|
1116
|
+
"""Evaluates pred against Synapse-Hub Gate 3 (Clean POM) and Gate 7 (infer_linter)."""
|
|
1117
|
+
score = 1.0
|
|
1118
|
+
if "xpath" in pred.test_script_code.lower() or "findtestobject" in pred.test_script_code.lower():
|
|
1119
|
+
return 0.0 # Violation of Clean POM invariant
|
|
1120
|
+
if not lint_fake_passes(pred.test_script_code):
|
|
1121
|
+
return 0.0 # Violation of Gate 7 (zero assertions or empty catch)
|
|
1122
|
+
return score
|
|
1123
|
+
```
|
|
1124
|
+
|
|
1125
|
+
### 6.5 Subsystem 4: Mutant-Directed Textual Loss Optimizer (TextGrad Pattern)
|
|
1126
|
+
|
|
1127
|
+
#### A. Architectural Design
|
|
1128
|
+
Connects `tools/micro_mutator.py` with a TextGrad backpropagation loop. When micro-mutators detect surviving mutants, the textual mutation report is backpropagated to compute a textual gradient $\nabla \mathcal{L}_{\text{Mutant}}$ that directly optimizes test generation prompts to ensure robust assertion synthesis.
|
|
1129
|
+
|
|
1130
|
+
#### B. Component Interface & Data Contract
|
|
1131
|
+
```python
|
|
1132
|
+
# tools/textgrad_mutation_optimizer.py
|
|
1133
|
+
import textgrad
|
|
1134
|
+
|
|
1135
|
+
class MutationLossEvaluator:
|
|
1136
|
+
def compute_loss(self, test_code: str) -> str:
|
|
1137
|
+
"""Runs tools/micro_mutator.py and formats surviving mutants into natural language loss."""
|
|
1138
|
+
result = run_micro_mutator(test_code)
|
|
1139
|
+
if result.mutation_score >= 80.0:
|
|
1140
|
+
return "SUCCESS: Mutation score satisfies DoD threshold (>= 80%)."
|
|
1141
|
+
return f"LOSS: Mutation score was {result.mutation_score}%. Surviving mutants: {result.surviving_summary}"
|
|
1142
|
+
|
|
1143
|
+
def optimize_test_prompt_with_textgrad(base_prompt_var, training_flows):
|
|
1144
|
+
optimizer = textgrad.TGD(parameters=[base_prompt_var])
|
|
1145
|
+
for flow in training_flows:
|
|
1146
|
+
optimizer.zero_grad()
|
|
1147
|
+
# Ensure generated_test retains predecessors=[base_prompt_var] to preserve DAG connectivity
|
|
1148
|
+
generated_test = llm_generate_test(base_prompt_var, flow)
|
|
1149
|
+
loss = MutationLossEvaluator().compute_loss(generated_test.value)
|
|
1150
|
+
if "LOSS" in loss:
|
|
1151
|
+
loss_var = textgrad.Variable(
|
|
1152
|
+
loss,
|
|
1153
|
+
role_description="Mutant survival loss",
|
|
1154
|
+
predecessors=[generated_test]
|
|
1155
|
+
)
|
|
1156
|
+
loss_var.backward()
|
|
1157
|
+
optimizer.step()
|
|
1158
|
+
```
|
|
1159
|
+
|
|
1160
|
+
### 6.6 Subsystem 5: Synapse-QA-Bench & Reversible Execution Sandbox (Gorilla & ToolBench Pattern)
|
|
1161
|
+
|
|
1162
|
+
#### A. Architectural Design
|
|
1163
|
+
1. **Reversible Execution Sandbox (GoEx Pattern)**: Wraps database seeding and file operations in `tools/safety_firewall.py`. Every `CREATE`, `INSERT`, or filesystem write logs an automated `REVERSE_ACTION` (e.g., `DELETE WHERE id = ...`) executed unconditionally in teardown.
|
|
1164
|
+
2. **Synapse-QA-Bench (BFCL Pattern)**: Expands `tools/agent_canary.py` into a standardized benchmark measuring tool selection accuracy, AST argument validation, and irrelevance detection across Synapse-Hub's 90 tools.
|
|
1165
|
+
|
|
1166
|
+
#### B. Component Interface & Data Contract
|
|
1167
|
+
```python
|
|
1168
|
+
# tools/reversible_sandbox.py
|
|
1169
|
+
@dataclass
|
|
1170
|
+
class ReversibleAction:
|
|
1171
|
+
action_type: str # "DB_INSERT", "FS_WRITE", "AUTH_SESSION"
|
|
1172
|
+
target_resource: str # Table name, file path, or session token
|
|
1173
|
+
undo_command: str # Exact SQL DELETE, rm, or revoke API call
|
|
1174
|
+
|
|
1175
|
+
class ExecutionSandbox:
|
|
1176
|
+
def __init__(self):
|
|
1177
|
+
self.undo_stack: List[ReversibleAction] = []
|
|
1178
|
+
|
|
1179
|
+
def execute_stateful_action(self, forward_cmd: str, reverse_cmd: str):
|
|
1180
|
+
"""Executes action and pushes reverse action onto teardown stack."""
|
|
1181
|
+
execute_forward(forward_cmd)
|
|
1182
|
+
self.undo_stack.append(ReversibleAction("STATE_MUTATION", forward_cmd, reverse_cmd))
|
|
1183
|
+
|
|
1184
|
+
def rollback_all(self):
|
|
1185
|
+
"""Guaranteed teardown execution."""
|
|
1186
|
+
while self.undo_stack:
|
|
1187
|
+
action = self.undo_stack.pop()
|
|
1188
|
+
execute_silent_undo(action.undo_command)
|
|
1189
|
+
```
|
|
1190
|
+
|
|
1191
|
+
### 6.7 Subsystem 6: Modern Packaging, Tier-2 Collision Guard & Grilling Primitive (AddyOsmani & MattPocock Pattern)
|
|
1192
|
+
|
|
1193
|
+
#### A. Architectural Design
|
|
1194
|
+
1. **Tier-2 Routing Collision Guard (Addy Pattern)**: Adds **Gate 10** to `tools/verify_all.py`. Uses an offline lightweight suffix-stripping stemmer heuristic and TF-IDF cosine similarity to verify that no two skills in `.agents/skills/` have description similarity $\ge 0.75$, preventing catalog ambiguity.
|
|
1195
|
+
2. **The Grilling Primitive (Pocock Pattern)**: Embedded into `orchestrator` during Stage 1. Calculates the prerequisite decision frontier, batches questions with recommended defaults, and respects the fact/decision boundary.
|
|
1196
|
+
3. **Universal CLI & Manifest Standards**: Adds `agentskills.io` metadata and Claude Code / Codex marketplace manifests to enable `npx skills add TeeVee998/Synapse-Hub`.
|
|
1197
|
+
|
|
1198
|
+
#### B. Component Interface: Tier-2 Collision Guard (Quality Gate 10)
|
|
1199
|
+
```python
|
|
1200
|
+
# tools/skill_collision_guard.py
|
|
1201
|
+
import re
|
|
1202
|
+
from typing import Dict, List, Tuple
|
|
1203
|
+
from math import sqrt
|
|
1204
|
+
from collections import Counter
|
|
1205
|
+
|
|
1206
|
+
class SkillCollisionGuard:
|
|
1207
|
+
def __init__(self, skills_dir: str = ".agents/skills"):
|
|
1208
|
+
self.skills_dir = skills_dir
|
|
1209
|
+
|
|
1210
|
+
def audit_skill_descriptions(self) -> Tuple[bool, List[str]]:
|
|
1211
|
+
"""Computes pairwise cosine similarity across all SKILL.md descriptions."""
|
|
1212
|
+
descriptions = self._load_descriptions()
|
|
1213
|
+
violations = []
|
|
1214
|
+
names = list(descriptions.keys())
|
|
1215
|
+
|
|
1216
|
+
for i in range(len(names)):
|
|
1217
|
+
for j in range(i + 1, len(names)):
|
|
1218
|
+
sim = self._cosine_sim(descriptions[names[i]], descriptions[names[j]])
|
|
1219
|
+
if sim >= 0.75:
|
|
1220
|
+
violations.append(f"CRITICAL COLLISION: '{names[i]}' and '{names[j]}' have similarity {sim:.2f} >= 0.75")
|
|
1221
|
+
elif sim >= 0.50:
|
|
1222
|
+
violations.append(f"WARNING: High similarity ({sim:.2f}) between '{names[i]}' and '{names[j]}'")
|
|
1223
|
+
|
|
1224
|
+
return len(violations) == 0, violations
|
|
1225
|
+
```
|
|
1226
|
+
|
|
1227
|
+
### 6.8 End-to-End Execution Sequence Flow
|
|
1228
|
+
|
|
1229
|
+
The following sequence illustrates how the enhanced subsystems collaborate during an autonomous test generation and execution cycle:
|
|
1230
|
+
|
|
1231
|
+
```mermaid
|
|
1232
|
+
sequenceDiagram
|
|
1233
|
+
autonumber
|
|
1234
|
+
actor User
|
|
1235
|
+
participant Orch as Orchestrator (Single Front-Door)
|
|
1236
|
+
participant Live as Live Explorer (Playwright)
|
|
1237
|
+
participant Vault as SkillVault (Voyager Pattern)
|
|
1238
|
+
participant Crafter as POM Crafter (DSPy Signature)
|
|
1239
|
+
participant Runner as Test Runner (GoEx Sandbox)
|
|
1240
|
+
participant Analyst as Failure Analyst (Reflexion Loop)
|
|
1241
|
+
participant Gate as Quality Gatekeeper (10 Gates DoD)
|
|
1242
|
+
|
|
1243
|
+
User->>Orch: "Test New Invoice Approval Workflow"
|
|
1244
|
+
Note over Orch: Pocock Grilling Primitive:<br/>Frontier Calculation & Defaults
|
|
1245
|
+
Orch->>Live: Probe Live DOM / AOM under 200ms
|
|
1246
|
+
Live-->>Orch: Compact AOM (<500 tokens)
|
|
1247
|
+
Orch->>Vault: Query Existing Invoice Page Classes
|
|
1248
|
+
Vault-->>Crafter: Retrieved Verified Methods (top-k=5)
|
|
1249
|
+
Note over Crafter: DSPy Compiled Signature:<br/>Synthesizes Clean POM + Flat Tests
|
|
1250
|
+
Crafter-->>Runner: Clean Test Code (Zero Raw XPaths)
|
|
1251
|
+
Note over Runner: GoEx Reversible Sandbox:<br/>Executes Headless & Pushes Undo Stack
|
|
1252
|
+
alt Test Execution Passes
|
|
1253
|
+
Runner->>Gate: Dispatches to Quality Gatekeeper
|
|
1254
|
+
else Test Execution Fails
|
|
1255
|
+
Runner->>Analyst: Dispatches JUnit XML & Screenshot
|
|
1256
|
+
loop Reflexion Loop (K=3)
|
|
1257
|
+
Analyst->>Analyst: Generate Verbal Reflection
|
|
1258
|
+
Analyst->>Crafter: Re-synthesize Patch with Reflection Context
|
|
1259
|
+
Crafter-->>Runner: Re-run Isolated Test
|
|
1260
|
+
end
|
|
1261
|
+
Runner->>Gate: Verified Healed Code
|
|
1262
|
+
end
|
|
1263
|
+
Note over Gate: Enforces 10 Quality Gates DoD:<br/>Gate 7 (infer_linter), Gate 8 (mutator >=80%),<br/>Gate 10 (Tier-2 Skill Collision <0.75)
|
|
1264
|
+
Gate-->>User: Verified Clean Commit + Teardown Executed
|
|
1265
|
+
```
|
|
1266
|
+
|
|
1267
|
+
---
|
|
1268
|
+
|
|
1269
|
+
## 7. Actionable Prioritized Integration Roadmap & Trade-Off Analysis
|
|
1270
|
+
|
|
1271
|
+
### 7.1 Prioritized Phased Implementation Plan (P0, P1, P2)
|
|
1272
|
+
|
|
1273
|
+
```
|
|
1274
|
+
2026 Q4 Roadmap Timeline:
|
|
1275
|
+
+-----------------------------------------------------------------------+
|
|
1276
|
+
| Phase P0 (Weeks 1-2): Core Hardening & Quality Gate Expansion |
|
|
1277
|
+
| - Gate 10: Tier-2 Skill Collision Guard in verify_all.py |
|
|
1278
|
+
| - GoEx Reversible Teardown Stack in safety_firewall.py |
|
|
1279
|
+
| - Anti-Rationalization Tables in Subagent Prompts |
|
|
1280
|
+
+-----------------------------------------------------------------------+
|
|
1281
|
+
| Phase P1 (Weeks 3-5): Reflexion Healing & Declarative Signatures |
|
|
1282
|
+
| - Closed-Loop Multi-Trial Reflexion (K=3) in auto_heal.py |
|
|
1283
|
+
| - DSPy Declarative Signatures for StoryToCleanPOM & RCA |
|
|
1284
|
+
| - Pocock Grilling Primitive in Orchestrator Discovery |
|
|
1285
|
+
| - DFSDT Backtracking Locator Fallbacks in browser_helpers.js |
|
|
1286
|
+
+-----------------------------------------------------------------------+
|
|
1287
|
+
| Phase P2 (Weeks 6-8): Autonomous SkillVault & TextGrad Optimization |
|
|
1288
|
+
| - SkillVault SQLite Library for Self-Evolving Page Classes |
|
|
1289
|
+
| - Offline TextGrad Prompt Optimizer driven by Micro-Mutator Loss |
|
|
1290
|
+
| - Synapse-QA-Bench 90-Tool Function Calling Benchmark (BFCL) |
|
|
1291
|
+
| - Modern Package Distribution (agentskills.io & npx skills) |
|
|
1292
|
+
+-----------------------------------------------------------------------+
|
|
1293
|
+
```
|
|
1294
|
+
|
|
1295
|
+
#### Phase P0: Immediate Safety Hardening & Quality Gate Expansion (Weeks 1–2)
|
|
1296
|
+
- **Objective**: Harden Synapse-Hub's verification gates and safety invariants with zero runtime token overhead.
|
|
1297
|
+
- **Deliverables**:
|
|
1298
|
+
1. *Quality Gate 10 (Tier-2 Skill Collision Guard)*: Integrate Addy Osmani's offline cosine similarity check into `tools/verify_all.py`. Fails pre-push if any two skills have description similarity $\ge 0.75$.
|
|
1299
|
+
2. *GoEx Reversible Teardown Stack*: Implement `ReversibleAction` stack in `tools/safety_firewall.py` guaranteeing automated rollback of database mutations and test files.
|
|
1300
|
+
3. *Anti-Rationalization Injection*: Inject Addy Osmani's anti-rationalization tables into `.agents/agents/pom-crafter.md`, `test-runner.md`, and `failure-analyst.md` to eliminate fake-pass excuses.
|
|
1301
|
+
|
|
1302
|
+
#### Phase P1: High-ROI Algorithmic Upgrades (Weeks 3–5)
|
|
1303
|
+
- **Objective**: Implement closed-loop self-healing, declarative prompt compilation, and requirements grilling.
|
|
1304
|
+
- **Deliverables**:
|
|
1305
|
+
1. *Multi-Trial Episodic Reflexion Loop*: Upgrade `tools/auto_heal.py` with an iterative reflection loop ($K=3$), persisting verbal diagnoses into `cache_resolver.py`.
|
|
1306
|
+
2. *DSPy Declarative QA Signatures*: Create `tools/dspy_qa_signatures.py` defining `UserStoryToUniversalIR` and `UniversalIRToCleanPOM`, compiled against Gate 3 and Gate 7 metrics.
|
|
1307
|
+
3. *Pocock Grilling Primitive*: Integrate design-tree frontier interrogation into `.agents/agents/orchestrator.md` and `tools/interview_story.py`.
|
|
1308
|
+
4. *DFSDT Backtracking Locator Resolver*: Implement Depth-First Search tree search in `tools/browser_helpers.js` to backtrack across the Master Locator Hierarchy upon stale element errors.
|
|
1309
|
+
|
|
1310
|
+
#### Phase P2: Autonomous Self-Evolution & Tool Benchmarking (Weeks 6–8)
|
|
1311
|
+
- **Objective**: Deploy lifelong skill libraries, textual gradient optimization, and standardized benchmarks.
|
|
1312
|
+
- **Deliverables**:
|
|
1313
|
+
1. *SkillVault Lifelong POM Library*: Deploy `.synapse/skill_vault.db` and `tools/skill_vault.py` for indexing verified Page Class methods and interaction primitives.
|
|
1314
|
+
2. *TextGrad Mutation-Loss Optimizer*: Create `tools/optimize_prompt_textgrad.py` to optimize test generation prompts using surviving mutant loss reports.
|
|
1315
|
+
3. *Synapse-QA-Bench (BFCL-Style Benchmark)*: Expand `tools/agent_canary.py` to benchmark all 90 tools on AST argument matching and irrelevance detection.
|
|
1316
|
+
4. *Modern Skill Packaging*: Add `agentskills.io` metadata and multi-host adapters to support `npx skills add TeeVee998/Synapse-Hub`.
|
|
1317
|
+
|
|
1318
|
+
---
|
|
1319
|
+
|
|
1320
|
+
### 7.2 Technical Trade-Off & Cost-Benefit Analysis
|
|
1321
|
+
|
|
1322
|
+
| Architectural Trade-Off | Chosen Strategy | Alternative Rejected | Concrete Engineering Rationale |
|
|
1323
|
+
|:---|:---|:---|:---|
|
|
1324
|
+
| **Compilation Token Cost vs. Runtime Efficiency** | **Offline Pre-Compilation (DSPy / TextGrad)** | Continuous Online Prompt Tuning | Offline compilation consumes tokens once during CI/CD builds, yielding frozen, zero-token prompt artifacts for production runs. Online tuning adds unacceptable latency and API expense to daily developer workflows. |
|
|
1325
|
+
| **Heuristic Single-Pass vs. Multi-Trial Reflexion** | **Multi-Trial Reflexion ($K=3$) with SQLite Memory** | Single-Shot Heuristic Retry | Single-shot retries repeat identical failure patterns when errors are non-trivial. Bounded reflection ($K=3$) achieves high first-trial accuracy gains while hard-capping token expenditure. |
|
|
1326
|
+
| **Monolithic Vector Store vs. Dual SQLite Cache** | **Dual SQLite Cache + BM25 (`cache_resolver.py`)** | Cloud Vector DB (Pinecone / Weaviate) | SQLite is zero-dependency, local-first, runs completely offline, and guarantees target enterprise NDA code never leaks to third-party vector clouds. |
|
|
1327
|
+
| **Free-Form Chat Interaction vs. Structured Grilling** | **Pocock Grilling Design Tree (Frontier Batching)** | Open-Ended Conversational Chat | Open-ended chat produces 5–10 roundtrips of conversational churn. Grilling batches prerequisite decisions with recommended defaults, settling scope in 1–2 turns. |
|
|
1328
|
+
| **Live External APIs vs. Sandboxed Reversible Execution** | **GoEx Reversible Execution + Network Mocking** | Live Production / Staging API Calls | Live API calls risk catastrophic irreversible mutations (deletions, billing charges). GoEx damage confinement and undo stacks guarantee zero residual contamination. |
|
|
1329
|
+
|
|
1330
|
+
---
|
|
1331
|
+
|
|
1332
|
+
### 7.3 Risk Matrix & Mitigation Strategies
|
|
1333
|
+
|
|
1334
|
+
| Risk Identifier | Severity | Likelihood | Impact Description | Mitigation Strategy |
|
|
1335
|
+
|:---:|:---:|:---:|:---|:---|
|
|
1336
|
+
| **R-01: Reflection Hallucination** | High | Medium | Reflexion generator invents non-existent DOM causes when error traces lack detail. | Condition reflection prompts strictly on compact AOM snapshots and verbatim JUnit stacktraces; reject reflections exceeding 300 tokens. |
|
|
1337
|
+
| **R-02: DSPy Compilation Drift** | Medium | Medium | Compiled prompts overfit to training instances, degrading on novel UI pages. | Enforce diverse train/dev splits (Auth, Tables, Modals, Forms) and run AST linters (`infer_linter.py`) across all evaluation steps. |
|
|
1338
|
+
| **R-03: SkillVault Bloat & Stale Selectors** | High | Low | Accumulating thousands of Page Class methods creates outdated or duplicate locators. | Implement TTL (Time-To-Live) and access-frequency pruning; invalidate primitives whose target AUT URL structure mutates. |
|
|
1339
|
+
| **R-04: TextGrad Gradient Instability** | Medium | High | Textual gradients overshoot or produce contradictory stylistic instructions. | Isolate TextGrad to offline batch prompt development; cap optimization steps ($N \le 5$); freeze base system instructions. |
|
|
1340
|
+
| **R-05: Docker Dependency for GoEx** | Medium | Low | Enterprise developer workstations lack Docker daemon permissions. | Implement dual-mode execution: full Docker containerization when daemon exists, falling back to process-level SQLite/FS transaction rollbacks. |
|
|
1341
|
+
|
|
1342
|
+
---
|
|
1343
|
+
|
|
1344
|
+
### 7.4 Definition of Done (DoD) & Acceptance Sign-Off
|
|
1345
|
+
|
|
1346
|
+
The integration of any pattern from this architectural blueprint into Synapse-Hub must strictly satisfy the following Definition of Done before merging:
|
|
1347
|
+
|
|
1348
|
+
- [ ] **Zero BOM & Path Portability**: All new files strictly 100% UTF-8 No BOM; zero absolute machine paths.
|
|
1349
|
+
- [ ] **Clean POM Discipline**: Zero raw XPaths or CSS selectors in test scripts; all locators encapsulated in Page Classes inheriting from `BasePage`.
|
|
1350
|
+
- [ ] **Quality Gate Verification**: Must achieve 100% PASS across all 10 Quality Gates via `python tools/verify_all.py`.
|
|
1351
|
+
- [ ] **Mutation Score Threshold**: New or modified test suites must achieve Mutation Score $\ge 80\%$ via `python tools/micro_mutator.py`.
|
|
1352
|
+
- [ ] **Zero Skill Collision**: Pairwise cosine similarity with existing skills must remain strictly $< 0.75$ via `tools/skill_collision_guard.py`.
|
|
1353
|
+
- [ ] **Teardown Guarantee**: Every state-mutating test action must register an automated undo callback in the `ReversibleAction` stack.
|
|
1354
|
+
- [ ] **Author Attribution & MIT License**: Unaltered preservation of copyright notice crediting **Thế Vị (@TeeVee998)** and repository provenance in compliance with the MIT License.
|
|
1355
|
+
|
|
1356
|
+
---
|
|
1357
|
+
*End of Technical Analysis & Architectural Blueprint — Synapse-Hub Enterprise QA Intelligence Platform*
|