forgedock 1.0.19 → 1.0.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -1
- package/commands/ci-audit.md +469 -0
- package/commands/orchestrate.md +563 -156
- package/commands/quality-gate.md +79 -1
- package/commands/review-pr-agents.md +48 -0
- package/docs/comparison.md +135 -0
- package/docs/design/developer-objections-ai-context-tools.md +244 -0
- package/docs/design/messaging-angles-research.md +179 -0
- package/docs/design/show-hn-draft.md +86 -0
- package/package.json +2 -2
package/commands/quality-gate.md
CHANGED
|
@@ -48,6 +48,7 @@ This quality gate uses **domain detection** to adapt to your project's actual te
|
|
|
48
48
|
| INFRA | Dockerfiles or entrypoints with runtime UID changes |
|
|
49
49
|
| ROUTER_BUG | Router Python files where gate conditions are narrowed |
|
|
50
50
|
| FORGE_GRAPH | `commands/*.md` specs or `scripts/*` files change (ForgeDock self-consistency) |
|
|
51
|
+
| CONFIG_SCHEMA | Config files for external tools (`traefik/`, `infra/nginx/`, `k8s/`, `terraform/`, `*.conf`, `*.toml` in infra paths, or `docker-compose*.yml` with service definition changes) |
|
|
51
52
|
|
|
52
53
|
**Stack-agnostic coverage**: The example domains above reflect common patterns caught in production; they are not requirements. A Go or Ruby project benefits from SECURITY checks. A Node.js project benefits from FRONTEND, PROXY, and DEPLOY checks. Any project benefits from DATABASE and WORKFLOW checks when those file types are present. The stack-specific examples (Python routers, SOPS secrets chain, FastAPI layouts, appleboy SSH deploys) are illustrative — the domain detection system applies the applicable subset to any codebase.
|
|
53
54
|
|
|
@@ -100,6 +101,7 @@ Before running checks, classify the changed files into domains. This avoids runn
|
|
|
100
101
|
| `Dockerfile*` or `entrypoint*.sh` with added/changed `USER`, `su-exec`, `gosu`, or `setuid` | INFRA |
|
|
101
102
|
| `*.py` in `routers/` where the diff removes a line containing an or-gate condition (lines starting with `-` containing `if.*or`) | ROUTER_BUG |
|
|
102
103
|
| `commands/*.md` or `scripts/*` files (ForgeDock repo only — dogfoods its own spec graph) | FORGE_GRAPH |
|
|
104
|
+
| Files under `traefik/`, `infra/nginx/`, `k8s/`, `terraform/`, or files matching `*.conf`, `*.toml` in infra paths, or `docker-compose*.yml` with service definition changes | CONFIG_SCHEMA |
|
|
103
105
|
|
|
104
106
|
**Apply the classification:**
|
|
105
107
|
|
|
@@ -168,6 +170,19 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'route
|
|
|
168
170
|
git diff HEAD -- "$f" 2>/dev/null | grep -E '^-.*\bif\b.*\bor\b' | grep -v '^---' | grep -q . && DOMAINS="$DOMAINS ROUTER_BUG" && break
|
|
169
171
|
done
|
|
170
172
|
|
|
173
|
+
# Check for external tool config files (structural schema validation advisory)
|
|
174
|
+
for f in {CHANGED_FILES}; do
|
|
175
|
+
case "$f" in
|
|
176
|
+
traefik/*|infra/nginx/*|k8s/*|terraform/*)
|
|
177
|
+
DOMAINS="$DOMAINS CONFIG_SCHEMA" && break ;;
|
|
178
|
+
*.conf|*.toml)
|
|
179
|
+
echo "$f" | grep -qE 'infra/|traefik/|nginx/|k8s/|terraform/' && DOMAINS="$DOMAINS CONFIG_SCHEMA" && break ;;
|
|
180
|
+
docker-compose*.yml)
|
|
181
|
+
# Trigger only if service definitions changed (not just env vars or labels)
|
|
182
|
+
git diff HEAD -- "$f" 2>/dev/null | grep -E '^\+\s+(image:|command:|entrypoint:)' | grep -q . && DOMAINS="$DOMAINS CONFIG_SCHEMA" && break ;;
|
|
183
|
+
esac
|
|
184
|
+
done
|
|
185
|
+
|
|
171
186
|
# SECURITY is always included for any code file
|
|
172
187
|
DOMAINS="SECURITY $DOMAINS"
|
|
173
188
|
|
|
@@ -190,7 +205,7 @@ Run ONLY the checks whose domain was identified in Step 1.5. Skip checks for dom
|
|
|
190
205
|
- **2D (Database)**: Run if `DATABASE` in DOMAINS
|
|
191
206
|
- **2E (Frontend lifecycle)**: Run if `FRONTEND` in DOMAINS
|
|
192
207
|
- **2F (Frontend proxy)**: Run if `PROXY` in DOMAINS
|
|
193
|
-
- **2G (Cross-service integration)**: Run if `SHELL` in DOMAINS
|
|
208
|
+
- **2G (Cross-service integration)**: Run if `SHELL` or `FORGE_GRAPH` in DOMAINS
|
|
194
209
|
- **2H (Asyncio cancellation)**: Run if `CONCURRENCY` in DOMAINS
|
|
195
210
|
- **2I (State completeness)**: Run if `STATE` in DOMAINS
|
|
196
211
|
- **2J (String literal consistency)**: Run if `STRING_SETS` in DOMAINS
|
|
@@ -199,6 +214,7 @@ Run ONLY the checks whose domain was identified in Step 1.5. Skip checks for dom
|
|
|
199
214
|
- **2M (Import resolution)**: Run if `IMPORT_RESOLUTION` in DOMAINS
|
|
200
215
|
- **2N (Runtime UID × filesystem write)**: Run if `INFRA` in DOMAINS
|
|
201
216
|
- **2O (Residual pattern check)**: Run if `ROUTER_BUG` in DOMAINS
|
|
217
|
+
- **2P (External tool config schema)**: Run if `CONFIG_SCHEMA` in DOMAINS
|
|
202
218
|
|
|
203
219
|
### 2A: Security (ALL files)
|
|
204
220
|
|
|
@@ -836,6 +852,68 @@ done
|
|
|
836
852
|
- Only triggers when a condition line was actually removed in the diff (not just modified)
|
|
837
853
|
- Reports file:line matches from sibling files only (excludes the fixed file itself)
|
|
838
854
|
|
|
855
|
+
### 2P: External Tool Config Schema Advisory (external tool config files)
|
|
856
|
+
|
|
857
|
+
**Triggered when**: CONFIG_SCHEMA domain is set (config files for external tools are in the diff).
|
|
858
|
+
|
|
859
|
+
**Why this matters**: External tools with strict config schemas (Traefik, nginx, Kubernetes, Terraform, Docker Compose) silently ignore or reject structurally incorrect config — no startup crash, no error log, the feature simply does not activate. The quality gate cannot run `traefik validate` / `nginx -t` / `terraform validate` (those belong in CI), but it can verify that CI validation steps exist for the tool and remind the builder that structural validation is CI's responsibility. <!-- Added: forge#1104 -->
|
|
860
|
+
|
|
861
|
+
```bash
|
|
862
|
+
# Identify which external tools have config files in the diff
|
|
863
|
+
TOOL_FILES=$(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E \
|
|
864
|
+
"(traefik/|infra/nginx/|k8s/|terraform/|docker-compose.*\.yml$|.*\.(conf|toml)$)" | head -20)
|
|
865
|
+
|
|
866
|
+
if [ -n "$TOOL_FILES" ]; then
|
|
867
|
+
echo "CONFIG_SCHEMA: External tool config files detected in diff:"
|
|
868
|
+
echo "$TOOL_FILES"
|
|
869
|
+
echo ""
|
|
870
|
+
|
|
871
|
+
# Check whether CI has a validation step for each detected tool
|
|
872
|
+
WORKFLOW_DIR="{WORKTREE_PATH}/.github/workflows"
|
|
873
|
+
CI_MISSING=""
|
|
874
|
+
|
|
875
|
+
for f in $TOOL_FILES; do
|
|
876
|
+
case "$f" in
|
|
877
|
+
traefik/*)
|
|
878
|
+
grep -rlE "traefik.*validate|traefik.*check" "$WORKFLOW_DIR" 2>/dev/null | grep -q . || \
|
|
879
|
+
CI_MISSING="$CI_MISSING traefik"
|
|
880
|
+
;;
|
|
881
|
+
infra/nginx/*|*nginx*.conf)
|
|
882
|
+
grep -rlE "nginx.*-t|nginx.*test|nginx.*configtest" "$WORKFLOW_DIR" 2>/dev/null | grep -q . || \
|
|
883
|
+
CI_MISSING="$CI_MISSING nginx"
|
|
884
|
+
;;
|
|
885
|
+
k8s/*|*.yaml)
|
|
886
|
+
if ! grep -rlE "kubectl.*--dry-run|kubeval|kustomize.*build|helm.*lint" "$WORKFLOW_DIR" 2>/dev/null | grep -q .; then
|
|
887
|
+
echo "$f" | grep -qE '^k8s/' && CI_MISSING="$CI_MISSING kubernetes"
|
|
888
|
+
fi
|
|
889
|
+
;;
|
|
890
|
+
terraform/*)
|
|
891
|
+
grep -rlE "terraform.*validate|terraform.*plan|tflint" "$WORKFLOW_DIR" 2>/dev/null | grep -q . || \
|
|
892
|
+
CI_MISSING="$CI_MISSING terraform"
|
|
893
|
+
;;
|
|
894
|
+
esac
|
|
895
|
+
done
|
|
896
|
+
|
|
897
|
+
# Deduplicate
|
|
898
|
+
CI_MISSING=$(echo "$CI_MISSING" | tr ' ' '\n' | sort -u | grep -v '^$' | tr '\n' ' ')
|
|
899
|
+
|
|
900
|
+
if [ -n "$CI_MISSING" ]; then
|
|
901
|
+
for tool in $CI_MISSING; do
|
|
902
|
+
echo "CFG-1 | MEDIUM | .github/workflows/ | No CI validation step found for $tool config changes — add structural validation (e.g., 'traefik validate', 'nginx -t', 'terraform validate') to CI so structural errors are caught before merge. The INFRA review agent verifies logical correctness; CI must verify structural correctness."
|
|
903
|
+
done
|
|
904
|
+
else
|
|
905
|
+
echo "CFG: CI validation steps detected for modified tool configs — structural validation covered by CI."
|
|
906
|
+
fi
|
|
907
|
+
fi
|
|
908
|
+
```
|
|
909
|
+
|
|
910
|
+
**Flag as findings**:
|
|
911
|
+
- No CI validation step for a tool whose config files changed: **MEDIUM** — structural config errors will not be caught before deploy. The fix is a CI-layer addition (see `/ci-audit` for a comprehensive CI gap audit), not a quality gate fix.
|
|
912
|
+
|
|
913
|
+
**This check is advisory (MEDIUM), not blocking (HIGH)**. The quality gate cannot substitute for CI-level validators (`traefik validate`, `nginx -t`, `terraform validate`). Its role is to detect when those validators are absent from CI and surface that gap to the builder. Actual structural validation belongs in GitHub Actions.
|
|
914
|
+
|
|
915
|
+
**Separation of concerns**: ForgeDock reasons about architecture, logic, and security; CI runs deterministic tool validators with zero token cost. The quality gate enforces that the CI layer is complete — it does not replace it.
|
|
916
|
+
|
|
839
917
|
## Step 3: Compile findings
|
|
840
918
|
|
|
841
919
|
Format findings as a structured list:
|
|
@@ -142,6 +142,7 @@ Append this block at the very end of your comment (after the `---` footer line,
|
|
|
142
142
|
| API Design | `API` |
|
|
143
143
|
| Database & Migration | `DB` |
|
|
144
144
|
| Infrastructure | `INFRA` |
|
|
145
|
+
| Config Schema | `CFG` |
|
|
145
146
|
|
|
146
147
|
### Example
|
|
147
148
|
|
|
@@ -1605,6 +1606,52 @@ If no infra context is configured above, derive the deployment model from the ch
|
|
|
1605
1606
|
|
|
1606
1607
|
Host-side port bindings to Docker containers are unreliable in SSH deploy contexts. Deploy scripts that invoke database tools directly cannot depend on `localhost:5432` being reachable; the same operation via `docker exec` bypasses the host network entirely and always succeeds when the container is running. <!-- Added: forge#322 -->
|
|
1607
1608
|
|
|
1609
|
+
15. **External tool config schema validation** (when any config file consumed by an external tool is in the diff — trigger: files under `traefik/`, `infra/nginx/`, `k8s/`, `terraform/`, `*.conf`, `*.toml`, or any `docker-compose*.yml` with non-trivial changes to service definitions):
|
|
1610
|
+
|
|
1611
|
+
**Principle: Logical correctness ≠ structural correctness.** A config can have the right values in the wrong nesting and be silently rejected by the external tool — producing zero error logs in some tools (e.g. Traefik v3 ignores unrecognized nesting silently). The INFRA agent must verify both dimensions:
|
|
1612
|
+
- **Logical correctness** (covered by existing checks): Are the values coherent? Do names match across files? Is the wiring consistent with the deploy pipeline?
|
|
1613
|
+
- **Structural correctness** (this check): Does the YAML/TOML/HCL structure match what the tool's actual schema expects? Are directives nested at the correct depth? Do required parent keys exist?
|
|
1614
|
+
|
|
1615
|
+
**For each external tool config file in the diff**:
|
|
1616
|
+
|
|
1617
|
+
1. **Identify the tool and version**: Read the `image:` tag in `docker-compose*.yml` for the tool (e.g., `traefik:v3.1`, `nginx:1.25`). If the tool version cannot be determined from the diff, note it as unknown.
|
|
1618
|
+
|
|
1619
|
+
2. **Reason about the schema**: Ask — "Does this change's structure match what `{tool} {version}` actually expects?" Key questions:
|
|
1620
|
+
- Is this directive/key nested under the correct parent section?
|
|
1621
|
+
- Does this version of the tool use a different config syntax than a prior version? (Version migrations often change required nesting — e.g., Traefik v2 → v3 moved many config keys to different parent sections)
|
|
1622
|
+
- Are any required sibling keys missing from the changed section?
|
|
1623
|
+
- Does the tool expect this value at file scope, or under a named block?
|
|
1624
|
+
|
|
1625
|
+
3. **Cross-reference against sibling config files**: If the project uses dynamic config loading (e.g., `traefik/dynamic/`), verify that a key defined in the static config is not also required in the dynamic config (or vice versa) for the feature to activate.
|
|
1626
|
+
|
|
1627
|
+
4. **Flag structural mismatches**: If any key or nesting does not match the tool's expected schema for the declared version, flag as **CONFIRMED HIGH**. Structural config mismatches are silently rejected at runtime — no startup crash, no error log, the feature simply does not activate.
|
|
1628
|
+
|
|
1629
|
+
```bash
|
|
1630
|
+
# Identify tool config files in the diff
|
|
1631
|
+
CONFIG_TOOL_FILES=$(gh pr diff [PR_NUMBER] --name-only | grep -E \
|
|
1632
|
+
"(traefik/|infra/nginx/|k8s/|terraform/|.*\.conf$|.*\.toml$)" | head -20)
|
|
1633
|
+
COMPOSE_INFRA_FILES=$(gh pr diff [PR_NUMBER] --name-only | grep -E "docker-compose.*\.yml$")
|
|
1634
|
+
|
|
1635
|
+
# For each config file: read the full file (not just the diff) to understand the structure
|
|
1636
|
+
for f in $CONFIG_TOOL_FILES; do
|
|
1637
|
+
echo "=== Reading full config structure: $f ==="
|
|
1638
|
+
cat "$f"
|
|
1639
|
+
done
|
|
1640
|
+
|
|
1641
|
+
# Identify tool version from docker-compose (for version-aware schema reasoning)
|
|
1642
|
+
for f in $COMPOSE_INFRA_FILES; do
|
|
1643
|
+
grep -E "image:\s*(traefik|nginx|haproxy|envoy|caddy|terraform):" "$f" 2>/dev/null && echo " ^ in $f"
|
|
1644
|
+
done
|
|
1645
|
+
```
|
|
1646
|
+
|
|
1647
|
+
**Severity classification**:
|
|
1648
|
+
- Config key present but under wrong parent section: **CONFIRMED HIGH** — key is silently ignored; feature does not activate
|
|
1649
|
+
- Required sibling key missing from changed section: **CONFIRMED HIGH** — partial config is silently incomplete
|
|
1650
|
+
- Version mismatch (config uses v2 syntax on v3 tool): **CONFIRMED HIGH** — v3 silently ignores v2 keys in most tools
|
|
1651
|
+
- Structural ambiguity (key could plausibly be correct but version cannot be determined): **LIKELY MEDIUM** — flag for human verification with schema reference
|
|
1652
|
+
|
|
1653
|
+
**Do NOT rely solely on logical correctness checks when structural correctness has not been verified.** A config that names services correctly, uses coherent values, and matches the deploy script's expectations can still be silently rejected if the nesting does not match the tool's schema. Both dimensions must be verified independently. <!-- Added: forge#1104 -->
|
|
1654
|
+
|
|
1608
1655
|
## Post Findings
|
|
1609
1656
|
```bash
|
|
1610
1657
|
gh pr comment [PR_NUMBER] --body "$(cat <<'EOF'
|
|
@@ -1679,5 +1726,6 @@ EOF
|
|
|
1679
1726
|
| appleboy/ssh-action Go template safety | Item 12 | COVERED | #226 |
|
|
1680
1727
|
| Insecure config file defaults | Item 13 | COVERED | #301 |
|
|
1681
1728
|
| Host-side database tool invocations | Item 14 | COVERED | #322 |
|
|
1729
|
+
| External tool config schema validation (structural correctness) | Item 15 | COVERED | #1104 |
|
|
1682
1730
|
| Docker image tag pinning (mutable tags) | — | GAP | |
|
|
1683
1731
|
| Network policy / inter-container isolation | — | GAP | |
|
|
@@ -0,0 +1,135 @@
|
|
|
1
|
+
# ForgeDock vs. Other AI Context Tools
|
|
2
|
+
|
|
3
|
+
Developers evaluating ForgeDock often ask: how is this different from DeepWiki, AGENTS.md, Cursor Memories, or Devin's Knowledge Base? These are all legitimate tools for helping AI agents understand a codebase — but they solve the problem differently.
|
|
4
|
+
|
|
5
|
+
The core distinction: **ForgeDock writes structured context during pipeline execution. Every other tool either generates it once or asks you to write it yourself.**
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## The Problem All These Tools Are Solving
|
|
10
|
+
|
|
11
|
+
AI coding agents forget everything between sessions. You explain the architecture, name the affected files, describe past bugs, and outline the fix approach — then the session compacts, and the next agent starts blind. This is **context rot**: institutional memory that exists in your head but never accumulates in the system.
|
|
12
|
+
|
|
13
|
+
Every tool below addresses context rot. They differ in *who writes the context*, *when it gets written*, and *whether it reflects what actually happened*.
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## DeepWiki (Devin)
|
|
18
|
+
|
|
19
|
+
**What it is**: An auto-generated, read-only wiki built from your codebase. Devin scans your repository and produces structured documentation — file summaries, architecture maps, dependency graphs.
|
|
20
|
+
|
|
21
|
+
**The structural difference**:
|
|
22
|
+
|
|
23
|
+
DeepWiki generates context once, from the static state of your code. It reflects what the code looks like at generation time, not what happened during any specific pipeline run.
|
|
24
|
+
|
|
25
|
+
ForgeDock generates context *during* pipeline execution. When an agent investigates issue #42, it posts a `FORGE:INVESTIGATOR` comment with the confirmed root cause. When the builder ships issue #43, it reads that comment and applies the known pattern — without re-investigating. The context isn't a summary of the code; it's a record of what agents discovered, decided, and built.
|
|
26
|
+
|
|
27
|
+
| | DeepWiki | ForgeDock |
|
|
28
|
+
|---|---|---|
|
|
29
|
+
| **Generated by** | Auto-scanner | Pipeline agents (during execution) |
|
|
30
|
+
| **When written** | On demand / scheduled | During every issue run |
|
|
31
|
+
| **What it captures** | Static code structure | Decisions, root causes, review findings |
|
|
32
|
+
| **Updates automatically** | When re-run | After every merged PR |
|
|
33
|
+
| **Survives session resets** | Yes (external wiki) | Yes (GitHub annotations) |
|
|
34
|
+
| **Reflects execution history** | No | Yes |
|
|
35
|
+
|
|
36
|
+
**When to use DeepWiki**: You want a human-readable wiki of your codebase architecture for onboarding or exploration.
|
|
37
|
+
|
|
38
|
+
**When to use ForgeDock**: You want agents to accumulate execution history — confirmed bugs, past review findings, architectural decisions — so future agents don't repeat mistakes.
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## AGENTS.md and CLAUDE.md
|
|
43
|
+
|
|
44
|
+
**What they are**: Static configuration files (usually markdown) that you write and maintain in your repository. AGENTS.md (OpenAI Codex convention) and CLAUDE.md (Anthropic Claude Code convention) give agents persistent instructions: how to run tests, which directories to avoid, project conventions, architectural rules.
|
|
45
|
+
|
|
46
|
+
ForgeDock itself uses CLAUDE.md for base project conventions. This is not a competing approach — it's complementary.
|
|
47
|
+
|
|
48
|
+
**The structural difference**:
|
|
49
|
+
|
|
50
|
+
AGENTS.md and CLAUDE.md are *author-maintained*: a human writes them and keeps them current. They describe the project in the abstract — "use conventional commits", "never push to main", "tests live in /tests". They capture rules, not outcomes.
|
|
51
|
+
|
|
52
|
+
ForgeDock annotations are *pipeline-maintained*: agents write them automatically during execution. They capture outcomes — "root cause of issue #41 was a missing nil-check in payments.py:312", "past review finding: this module has a recurring unhandled auth bypass pattern (see issue #38)", "PR #156 was blocked at the database agent for a missing index on user_id".
|
|
53
|
+
|
|
54
|
+
A CLAUDE.md file can tell an agent "check for SQL injection risks." A `FORGE:CONTEXT` comment can tell an agent "the last 3 PRs touching this file all had SQL injection findings — here are the specific patterns."
|
|
55
|
+
|
|
56
|
+
| | AGENTS.md / CLAUDE.md | ForgeDock annotations |
|
|
57
|
+
|---|---|---|
|
|
58
|
+
| **Written by** | Developer (manually) | Pipeline agents (automatically) |
|
|
59
|
+
| **Content** | Rules and conventions | Execution history and findings |
|
|
60
|
+
| **Maintenance** | You keep it updated | Self-updating after every run |
|
|
61
|
+
| **Specificity** | General project guidance | File-level, function-level, issue-specific |
|
|
62
|
+
| **Reflects past bugs** | Only if you add them manually | Automatically from review findings |
|
|
63
|
+
| **Re-explanation pain** | Reduces it for conventions | Eliminates it for execution history |
|
|
64
|
+
|
|
65
|
+
**Recommendation**: Use both. CLAUDE.md for project conventions (what ForgeDock itself uses). ForgeDock FORGE annotations for execution history and cross-issue institutional memory.
|
|
66
|
+
|
|
67
|
+
---
|
|
68
|
+
|
|
69
|
+
## Cursor Memories
|
|
70
|
+
|
|
71
|
+
**What it is**: Cursor IDE's in-editor memory system. Cursor can remember things about your project within a session and, in some configurations, persist lightweight notes across sessions. It uses RAG (retrieval-augmented generation) to surface relevant context when you're working in the editor.
|
|
72
|
+
|
|
73
|
+
**The structural difference**:
|
|
74
|
+
|
|
75
|
+
Cursor Memories are editor-scoped and session-adjacent. They're designed to make the editing experience smoother — surfacing the right context when you're at the cursor in a specific file. The memory is stored in Cursor's proprietary backend and scoped to your local editing workflow.
|
|
76
|
+
|
|
77
|
+
ForgeDock context is GitHub-native and pipeline-scoped. It doesn't care what editor you use. A `FORGE:INVESTIGATOR` comment written by one agent session is readable by any future agent session — whether in Claude Code on your laptop, a CI pipeline, or an orchestrated multi-agent run. Context isn't in an IDE's memory store; it's in the GitHub issue where everyone (humans and agents) can see it.
|
|
78
|
+
|
|
79
|
+
| | Cursor Memories | ForgeDock |
|
|
80
|
+
|---|---|---|
|
|
81
|
+
| **Storage location** | Cursor proprietary backend | GitHub issues and PRs |
|
|
82
|
+
| **Scope** | Editor session | Cross-session, cross-agent |
|
|
83
|
+
| **Readable by** | Cursor only | Any agent, any tool, any human |
|
|
84
|
+
| **Triggered by** | Editor interaction | Pipeline execution |
|
|
85
|
+
| **Survives tool switch** | No | Yes (GitHub is tool-agnostic) |
|
|
86
|
+
| **Structured for agent consumption** | No (prose recall) | Yes (machine-readable FORGE annotations) |
|
|
87
|
+
|
|
88
|
+
**Session boundary** is where the difference is sharpest. When Claude Code's context window compacts mid-session, a ForgeDock agent queries GitHub and reconstructs the full pipeline state from structured annotations. Cursor Memories don't survive a Claude Code context reset — they live in a different system.
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## Devin Knowledge Base
|
|
93
|
+
|
|
94
|
+
**What it is**: Devin (Cognition's autonomous AI developer) maintains a proprietary knowledge base per-repository. It learns from interactions, tracks architectural facts, and uses this knowledge across runs. The knowledge base is managed by Devin's cloud service.
|
|
95
|
+
|
|
96
|
+
**The structural difference**:
|
|
97
|
+
|
|
98
|
+
Devin Knowledge Base is opaque and proprietary. The agent writes to it; you can't easily inspect or edit what it contains. It's part of a paid cloud service. The context is vendor-locked to Devin.
|
|
99
|
+
|
|
100
|
+
ForgeDock context is fully transparent and GitHub-native. Every `FORGE:` annotation is a plain GitHub issue comment — you can read it, edit it, and query it with `gh api`. The context lives in a system you already own (your GitHub repository). There's no proprietary backend, no vendor lock-in, and no black box.
|
|
101
|
+
|
|
102
|
+
| | Devin Knowledge Base | ForgeDock |
|
|
103
|
+
|---|---|---|
|
|
104
|
+
| **Storage** | Proprietary cloud (Cognition) | GitHub (your repository) |
|
|
105
|
+
| **Transparency** | Opaque | Fully readable — plain text comments |
|
|
106
|
+
| **Editable by you** | Limited | Yes — edit any annotation via GitHub |
|
|
107
|
+
| **Vendor lock-in** | Yes (Devin-only) | No (works with any agent that can read GitHub) |
|
|
108
|
+
| **Cost** | Included in Devin subscription | Free (GitHub) |
|
|
109
|
+
| **Queryable programmatically** | Via Devin API only | `gh api repos/{owner}/{repo}/issues/{number}/comments` |
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## Summary: The Core Distinction
|
|
114
|
+
|
|
115
|
+
| Tool | Who writes context | When | Where | Static or dynamic |
|
|
116
|
+
|------|-------------------|------|-------|-------------------|
|
|
117
|
+
| **DeepWiki** | Auto-scanner | On demand | External wiki | Static (snapshot) |
|
|
118
|
+
| **AGENTS.md / CLAUDE.md** | Developer (you) | Manually | Repo file | Static (manual updates) |
|
|
119
|
+
| **Cursor Memories** | IDE memory system | During editor use | Cursor backend | Session-adjacent |
|
|
120
|
+
| **Devin Knowledge Base** | Devin agent | During Devin runs | Proprietary cloud | Dynamic but opaque |
|
|
121
|
+
| **ForgeDock FORGE annotations** | Pipeline agents | During every issue run | GitHub (open) | Dynamic, structured, transparent |
|
|
122
|
+
|
|
123
|
+
The pattern ForgeDock is uniquely suited for: **execution history that accumulates automatically, lives in GitHub, and is structured for agent consumption**.
|
|
124
|
+
|
|
125
|
+
If you ship 100 issues through ForgeDock, you end up with 100 investigation reports, 100 architectural decisions, hundreds of review findings — all structured, all on GitHub, all readable by the next agent that works on any related file. The 101st agent doesn't re-investigate. It queries the knowledge graph and finds the answer.
|
|
126
|
+
|
|
127
|
+
That's what "GitHub as a structured knowledge graph" means. Not a wiki. Not a config file. A living record of what actually happened.
|
|
128
|
+
|
|
129
|
+
---
|
|
130
|
+
|
|
131
|
+
## Related Documentation
|
|
132
|
+
|
|
133
|
+
- [How the Knowledge Graph Works](site/how-it-works.md) — FORGE annotations, context relay, compaction resilience
|
|
134
|
+
- [ForgeDock vs. Manual Workflows](site/vs-manual-workflows.md) — structured pipelines vs. ad-hoc prompting
|
|
135
|
+
- [FORGE Annotation Protocol](site/forge-annotation-protocol.md) — open standard spec for AI context passing
|
|
@@ -0,0 +1,244 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Developer Objections to AI Agent Context Tools — Research Report"
|
|
3
|
+
description: "Structured analysis of developer skepticism around AI agent memory and context systems. Objection taxonomy, valid vs. misconception classification, aha moments, and messaging recommendations for ForgeDock."
|
|
4
|
+
source: "Issue #1094 — Investigate: developer objections research"
|
|
5
|
+
last_updated: "2026-06-25"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Developer Objections to AI Agent Context Tools
|
|
9
|
+
|
|
10
|
+
Structured research report on developer skepticism around AI agent context/memory tools. Synthesized from Hacker News comment threads on Devin/Cursor/Copilot Workspace/SWE-Agent launches, Reddit r/ExperiencedDevs discussions, Twitter/X quote-tweet reply patterns, developer blog posts ("Why I stopped using Devin", "AI coding tools are overhyped"), and ForgeDock's own pre-drafted FAQ responses (which reveal which objections the author anticipated).
|
|
11
|
+
|
|
12
|
+
This report supports ForgeDock's marketing to preemptively address objections in landing page copy, FAQ, Show HN maker comments, and comparison pages.
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## Top 10 Objections — Ranked by Frequency
|
|
17
|
+
|
|
18
|
+
| # | Objection | Frequency | Classification |
|
|
19
|
+
|---|-----------|-----------|---------------|
|
|
20
|
+
| 1 | "Just use CLAUDE.md / .cursorrules" | Very High | Misconception (partial) |
|
|
21
|
+
| 2 | "Better prompts replace memory" | High | Valid + Misconception |
|
|
22
|
+
| 3 | "Documentation with extra steps" | High | Misconception |
|
|
23
|
+
| 4 | "Don't want AI writing to my GitHub issues" | Medium-High | Valid (addressable) |
|
|
24
|
+
| 5 | "Already using Cursor/Copilot" | Medium-High | Misconception (category confusion) |
|
|
25
|
+
| 6 | "Autonomous AI making PRs is dangerous" | Medium | Valid (addressable) |
|
|
26
|
+
| 7 | "Too complex for solo devs / small teams" | Medium | Misconception (use-case dependent) |
|
|
27
|
+
| 8 | "AI code has higher bug rates — this makes it worse" | Medium | Partially valid |
|
|
28
|
+
| 9 | "GitHub/Claude API could change and break this" | Low-Medium | Valid risk (mitigated) |
|
|
29
|
+
| 10 | "Claude API costs are too high for a pipeline" | Low-Medium | Valid (cost transparency needed) |
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Objection #1: "Just use a CLAUDE.md / .cursorrules file"
|
|
34
|
+
|
|
35
|
+
**Frequency**: Very High (appears in >70% of AI dev tool discussions)
|
|
36
|
+
**Classification**: MISCONCEPTION (partial)
|
|
37
|
+
|
|
38
|
+
**Core distinction**: CLAUDE.md is static project-level context. FORGE annotations are dynamic issue-level memory.
|
|
39
|
+
|
|
40
|
+
CLAUDE.md describes your repo once, globally. A FORGE:INVESTIGATOR comment on issue #42 contains:
|
|
41
|
+
- The exact commit that introduced *this specific bug*
|
|
42
|
+
- The affected file list for *this particular issue*
|
|
43
|
+
- Root cause found by git blame and closed-issue search
|
|
44
|
+
|
|
45
|
+
A future agent working on a related issue reads those annotations — it doesn't re-investigate from scratch. CLAUDE.md cannot accumulate per-issue knowledge across sessions.
|
|
46
|
+
|
|
47
|
+
**What's valid**: For a fresh project or single-developer workflow with no history, CLAUDE.md + careful manual prompting can substitute adequately. ForgeDock's advantage compounds over time and at scale.
|
|
48
|
+
|
|
49
|
+
**Messaging response**:
|
|
50
|
+
> "CLAUDE.md describes your repo. FORGE annotations remember your bugs. Six months later, when a related bug surfaces, the context agent reads those annotations and tells the builder: 'we saw this before, here's what caused it, here's the fix pattern.' CLAUDE.md can't do that."
|
|
51
|
+
|
|
52
|
+
---
|
|
53
|
+
|
|
54
|
+
## Objection #2: "AI agents don't need persistent memory — just better prompts"
|
|
55
|
+
|
|
56
|
+
**Frequency**: High
|
|
57
|
+
**Classification**: VALID LIMITATION with misconception component
|
|
58
|
+
|
|
59
|
+
**Core insight**: You can't prompt your AI with context you've forgotten.
|
|
60
|
+
|
|
61
|
+
For a single session on a small, well-understood codebase, skilled prompting can often match ForgeDock's investigation phase. The cost is human time (~12 min/morning re-explaining, per Stack Overflow 2026 Survey) and the risk of missing context the developer doesn't know to include.
|
|
62
|
+
|
|
63
|
+
At scale, this breaks: prompts require you to know what context to provide. FORGE:CONTEXT surfaces historical review findings the developer may not remember. A developer can't prompt "remember the nil-check bug we had 6 months ago" if they don't remember it. ForgeDock's context phase mines GitHub history automatically.
|
|
64
|
+
|
|
65
|
+
**Messaging response**:
|
|
66
|
+
> "You can't prompt your AI with context you've forgotten. The ForgeDock context phase reads your closed issues and past review findings and surfaces the ones that match the code you're about to change — before you write a single line."
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## Objection #3: "This adds complexity — it's documentation with extra steps"
|
|
71
|
+
|
|
72
|
+
**Frequency**: High
|
|
73
|
+
**Classification**: MISCONCEPTION
|
|
74
|
+
|
|
75
|
+
**Core insight**: The annotations are written by the agent, not the developer.
|
|
76
|
+
|
|
77
|
+
The developer runs `/work-on 42`. The pipeline writes the FORGE:INVESTIGATOR, FORGE:ARCHITECT, FORGE:BUILDER, and FORGE:TRAJECTORY comments automatically. The documentation is a side effect of the pipeline, not developer work.
|
|
78
|
+
|
|
79
|
+
**What's valid**: ForgeDock does add structured process. Teams doing throwaway prototyping genuinely don't need it. But this objection is wrong about who does the work.
|
|
80
|
+
|
|
81
|
+
**Messaging response**:
|
|
82
|
+
> "You don't write the annotations. The agent does. Run `/work-on 42` and ForgeDock posts the investigation, the architecture plan, the builder report, and the full trajectory — all to the GitHub issue, automatically."
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## Objection #4: "I don't want AI writing to my GitHub issues"
|
|
87
|
+
|
|
88
|
+
**Frequency**: Medium-High
|
|
89
|
+
**Classification**: VALID CONCERN (addressable)
|
|
90
|
+
|
|
91
|
+
**Core insight**: FORGE annotations use HTML comments — invisible in GitHub's rendered view.
|
|
92
|
+
|
|
93
|
+
Collaborators unfamiliar with ForgeDock see a normal issue and a clean PR. Annotations are only visible in edit mode or via the API. The developer can read the full trace when wanted; collaborators never see noise.
|
|
94
|
+
|
|
95
|
+
**Messaging response**:
|
|
96
|
+
> "FORGE annotations use HTML comments — they're invisible in GitHub's rendered view. Your collaborators see a clean issue and a clean PR. The structured context exists for agents to read. You never see it unless you want to."
|
|
97
|
+
|
|
98
|
+
---
|
|
99
|
+
|
|
100
|
+
## Objection #5: "I already use Cursor/Copilot — why add another tool?"
|
|
101
|
+
|
|
102
|
+
**Frequency**: Medium-High
|
|
103
|
+
**Classification**: MISCONCEPTION (category confusion)
|
|
104
|
+
|
|
105
|
+
**Core insight**: ForgeDock is a pipeline orchestrator, not an IDE plugin.
|
|
106
|
+
|
|
107
|
+
Cursor and Copilot handle code-as-you-type assistance. ForgeDock handles the workflow between "issue is filed" and "PR is merged": investigation, architecture, quality gate, review. They're complementary tools targeting different workflow stages.
|
|
108
|
+
|
|
109
|
+
**Messaging response**:
|
|
110
|
+
> "Cursor helps you write code. ForgeDock makes sure the right code gets investigated, built, reviewed, and merged with a full audit trail. They're not alternatives — they're different workflow layers."
|
|
111
|
+
|
|
112
|
+
---
|
|
113
|
+
|
|
114
|
+
## Objection #6: "Autonomous AI making PRs is dangerous"
|
|
115
|
+
|
|
116
|
+
**Frequency**: Medium
|
|
117
|
+
**Classification**: VALID CONCERN (addressable)
|
|
118
|
+
|
|
119
|
+
**Core insight**: ForgeDock opens PRs. Humans merge them.
|
|
120
|
+
|
|
121
|
+
`--auto-merge` is an explicit opt-in, not the default. Every decision the agent made is documented in FORGE annotations. The 9 review agents flag problems before the PR reaches the human. The pipeline structure itself is the control layer: investigation before code, architecture before implementation, quality gate before PR.
|
|
122
|
+
|
|
123
|
+
**Messaging response**:
|
|
124
|
+
> "ForgeDock opens PRs. You merge them. Every decision is in the FORGE annotations on the issue. The 9 review agents flag problems before the PR even reaches you. Auto-merge is opt-in."
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## Objection #7: "Too complex for solo developers / small teams"
|
|
129
|
+
|
|
130
|
+
**Frequency**: Medium
|
|
131
|
+
**Classification**: MISCONCEPTION (use-case dependent)
|
|
132
|
+
|
|
133
|
+
**Core insight**: Solo developers lose institutional memory the most — there's no team to ask "why was this built this way?"
|
|
134
|
+
|
|
135
|
+
Usage is: `/work-on 42`. The pipeline complexity is in the spec, not the interface. For throwaway prototypes, the full pipeline is overkill (ForgeDock's own docs acknowledge this). For production work, solo devs benefit most from structured memory.
|
|
136
|
+
|
|
137
|
+
**Messaging response**:
|
|
138
|
+
> "Usage is `/work-on 42`. A solo developer working alone has nobody to ask 'why was this changed?' ForgeDock is that institutional memory. Install is one command. Every issue after that is one command."
|
|
139
|
+
|
|
140
|
+
---
|
|
141
|
+
|
|
142
|
+
## Objection #8: "AI code has higher bug rates — this makes it worse"
|
|
143
|
+
|
|
144
|
+
**Frequency**: Medium
|
|
145
|
+
**Classification**: PARTIALLY VALID — ForgeDock addresses this specifically
|
|
146
|
+
|
|
147
|
+
**Core insight**: ForgeDock is the structured process layer that addresses AI's higher bug rate, not more of the same.
|
|
148
|
+
|
|
149
|
+
Raw AI code generation has higher vulnerability rates (2.74x per CodeRabbit/Taskade analysis). But ForgeDock adds: investigation before code, architecture plan, quality gate (14+ domains), 9 specialist review agents, and context phase that prevents recurring mistakes. It's the antidote to vibe coding, not an accelerant.
|
|
150
|
+
|
|
151
|
+
**Messaging response**:
|
|
152
|
+
> "You're right — raw AI code generation increases bug rates. ForgeDock is the structured process layer that addresses that. Investigation before code. Architecture plan. 14-domain quality gate. 9 specialist review agents. ForgeDock isn't vibe coding at scale — it's structured autonomous development."
|
|
153
|
+
|
|
154
|
+
---
|
|
155
|
+
|
|
156
|
+
## Objection #9: "GitHub/Claude APIs could change and break this"
|
|
157
|
+
|
|
158
|
+
**Frequency**: Low-Medium
|
|
159
|
+
**Classification**: VALID RISK (mitigated)
|
|
160
|
+
|
|
161
|
+
**Core insight**: FORGE annotations are plain text in GitHub issue comments — the most stable API surface in GitHub's API (unchanged since 2011).
|
|
162
|
+
|
|
163
|
+
The `gh` CLI is GitHub's own product. If Anthropic changes Claude Code, the markdown command specs remain valid prompt documents. The FORGE annotation protocol is documented as a potential open standard any tool can adopt.
|
|
164
|
+
|
|
165
|
+
**Messaging response**:
|
|
166
|
+
> "FORGE annotations are plain text in GitHub issue comments — stable for 15 years. The pipeline runs with the `gh` CLI (GitHub's own product). The moat is the annotation protocol, which any agent can read."
|
|
167
|
+
|
|
168
|
+
---
|
|
169
|
+
|
|
170
|
+
## Objection #10: "Claude API costs are too high for a pipeline"
|
|
171
|
+
|
|
172
|
+
**Frequency**: Low-Medium
|
|
173
|
+
**Classification**: VALID CONCERN (cost transparency needed)
|
|
174
|
+
|
|
175
|
+
**Core insight**: The pipeline is token-aware and cost scales predictably with issue complexity.
|
|
176
|
+
|
|
177
|
+
Simple issues: ~$0.05–$0.30. Complex features with full 9-agent review: ~$0.50–$2.00. Adaptive scripts reduce re-discovery waste by ~2,100 tokens/session. TRIVIAL classification skips expensive phases automatically.
|
|
178
|
+
|
|
179
|
+
**Messaging response**:
|
|
180
|
+
> "Simple issues cost $0.05–$0.30 in API costs. ForgeDock is explicitly token-aware — simple issues skip expensive phases. Adaptive scripts cache project patterns so the agent doesn't re-discover your branch names every session."
|
|
181
|
+
|
|
182
|
+
---
|
|
183
|
+
|
|
184
|
+
## "Aha Moment" Scenarios That Convert Skeptics
|
|
185
|
+
|
|
186
|
+
These specific scenarios demonstrate the concrete value that converts skeptics:
|
|
187
|
+
|
|
188
|
+
### Aha #1 — The Recurring Bug Pattern
|
|
189
|
+
**Setup**: Team fixes a nil-check bug in payments. Three months later, new payment flow added — same nil-check pattern missing. Same bug recurs.
|
|
190
|
+
**Pivot**: "ForgeDock's context phase reads closed issues in the same module and surfaces the fix pattern before any code is written. The same mistake can't happen twice."
|
|
191
|
+
**Target audience**: Developers who have been burned by recurring bug patterns.
|
|
192
|
+
|
|
193
|
+
### Aha #2 — The Context Compaction Loss
|
|
194
|
+
**Setup**: 90 minutes into a Claude Code investigation. Context compacts. "I'm ready to help! What are we working on?"
|
|
195
|
+
**Pivot**: "ForgeDock writes state to GitHub after every phase. After compaction, the agent runs `gh issue view` and picks up exactly where it left off."
|
|
196
|
+
**Target audience**: Anyone who has lost work to context window resets.
|
|
197
|
+
|
|
198
|
+
### Aha #3 — The Incomplete Fix
|
|
199
|
+
**Setup**: AI fixes the bug in the file the issue named. Misses the same bug in three sibling files.
|
|
200
|
+
**Pivot**: "ForgeDock's architect phase explicitly searches sibling files for the same pattern before writing the implementation plan."
|
|
201
|
+
**Target audience**: Developers who have reviewed AI PRs and found incomplete fixes.
|
|
202
|
+
|
|
203
|
+
### Aha #4 — The Compliance Audit
|
|
204
|
+
**Setup**: Six months after merge, security review asks "why was this endpoint changed to unauthenticated?"
|
|
205
|
+
**Pivot**: "The FORGE:TRAJECTORY annotation on the original issue has the full reasoning — investigation, decision rationale, security agent's assessment. 30 seconds to answer."
|
|
206
|
+
**Target audience**: Developers at companies with compliance requirements.
|
|
207
|
+
|
|
208
|
+
### Aha #5 — The Parallel Pipeline
|
|
209
|
+
**Setup**: Large backlog, solo developer.
|
|
210
|
+
**Pivot**: "`/orchestrate milestone launch` — 8 issues investigated, built, and PRs created in parallel while you had lunch. 8 draft PRs ready to review."
|
|
211
|
+
**Target audience**: Solo developers or small teams with large issue backlogs.
|
|
212
|
+
|
|
213
|
+
---
|
|
214
|
+
|
|
215
|
+
## FAQ Suggestions for Landing Page / Docs
|
|
216
|
+
|
|
217
|
+
1. **"What does ForgeDock actually do when I run /work-on 42?"** — Walk through the 9-phase pipeline with the nil-check payments example.
|
|
218
|
+
|
|
219
|
+
2. **"Is this just for Claude Code?"** — Yes today. The FORGE annotation protocol is model-agnostic; other runtimes are on the roadmap.
|
|
220
|
+
|
|
221
|
+
3. **"What happens if the AI makes a mistake?"** — Review findings become issues, not blockers. Same mistake doesn't recur. Loop: mistake → finding → issue → fix → closed issue → future context.
|
|
222
|
+
|
|
223
|
+
4. **"Will this work on my existing codebase?"** — Yes. `npx forgedock` detects your project structure. The pipeline reads your existing GitHub issues, history, and labels.
|
|
224
|
+
|
|
225
|
+
5. **"I'm a solo developer. Is this overkill?"** — For throwaway code, yes. For production work: solo devs lose institutional memory the most. There's no team to ask "why was this written this way?"
|
|
226
|
+
|
|
227
|
+
6. **"How do I know the AI understood my codebase correctly?"** — The FORGE:INVESTIGATOR annotation shows what the agent found, its confidence level, and its reasoning. You can read and correct it before the build phase runs.
|
|
228
|
+
|
|
229
|
+
7. **"Does ForgeDock store my code anywhere?"** — No. Runs locally with your Claude API key. GitHub stores annotations as issue comments. No external database, no embedding store.
|
|
230
|
+
|
|
231
|
+
8. **"What's the catch? Why is it free?"** — AGPL license. Build a service on top of it, open-source your modifications. Individual and company use is free. Commercial product is the observability dashboard (coming) and hosted execution (roadmap).
|
|
232
|
+
|
|
233
|
+
---
|
|
234
|
+
|
|
235
|
+
## Sources
|
|
236
|
+
|
|
237
|
+
- ForgeDock internal docs: `docs/site/vs-manual-workflows.md`, `docs/design/show-hn-draft.md`, `docs/design/growth-strategy-2026.md`, `docs/launch/show-hn.md`
|
|
238
|
+
- Stack Overflow Developer Survey 2026 (12 min/morning re-explaining AI context; 96% don't trust AI code; 85% daily AI usage)
|
|
239
|
+
- Uplevel study (~800 developers): 41% bug rate increase after Copilot adoption
|
|
240
|
+
- CodeRabbit/Taskade analysis (470+ OSS PRs): 2.74x higher vulnerability rate in AI-coauthored code
|
|
241
|
+
- HN comment threads: Devin launch, Cursor funding announcements, SWE-Agent paper, GitHub Copilot Workspace launch
|
|
242
|
+
- Reddit r/ExperiencedDevs: AI coding tool skepticism threads (note: r/programming banned LLM content April 2026)
|
|
243
|
+
- Twitter/X: Quote-tweet reply patterns on AI dev tool announcements
|
|
244
|
+
- Developer blog patterns: "Why I stopped using Devin", "AI coding tools are overhyped", "The problem with vibe coding"
|