forgedock 1.0.20 → 1.0.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -1
- package/commands/ci-audit.md +469 -0
- package/commands/orchestrate.md +365 -8
- package/commands/quality-gate.md +79 -1
- package/commands/review-pr-agents.md +48 -0
- package/docs/comparison.md +135 -0
- package/docs/design/developer-objections-ai-context-tools.md +244 -0
- package/docs/design/messaging-angles-research.md +179 -0
- package/docs/design/show-hn-draft.md +86 -0
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -55,7 +55,7 @@ ForgeDock is **not another AI coding agent.** It's a set of prompt-engineered co
|
|
|
55
55
|
| **Full-lifecycle automation** | `/work-on #42` — investigates the issue, architects a fix, builds it, runs quality gates, opens a PR, and reviews it. You click merge. |
|
|
56
56
|
| **Persistent agent memory** | Structured `FORGE:` annotations on GitHub issues/PRs survive context resets and session boundaries. Agents never start blind. |
|
|
57
57
|
| **9 specialist review agents** | Security, billing, database, concurrency, auth, frontend, API, performance, infrastructure — every PR gets domain-expert review. |
|
|
58
|
-
| **
|
|
58
|
+
| **Institutional memory that compounds** | Agent fixing issue #43 reads the investigation from #42 and applies the known pattern — no re-investigation. |
|
|
59
59
|
| **Self-improving pipeline** | Review agents learn from past findings — recurring patterns automatically become new quality gate checks. |
|
|
60
60
|
| **Parallel orchestration** | `/orchestrate` decomposes milestones into waves and runs `/work-on` on each in parallel. |
|
|
61
61
|
|
|
@@ -223,11 +223,26 @@ Removes all ForgeDock command symlinks from `~/.claude/commands/`. Your `forge.y
|
|
|
223
223
|
|
|
224
224
|
---
|
|
225
225
|
|
|
226
|
+
## FAQ — How is ForgeDock different from X?
|
|
227
|
+
|
|
228
|
+
**vs. DeepWiki (Devin)** — DeepWiki generates a read-only wiki from your codebase once. ForgeDock's FORGE annotations are written by pipeline agents during every issue run — capturing root causes, architectural decisions, and review findings as they happen, not as a snapshot of static code.
|
|
229
|
+
|
|
230
|
+
**vs. AGENTS.md / CLAUDE.md** — These files are written and maintained by you. They capture conventions and rules. ForgeDock annotations are written automatically by pipeline agents and capture execution history — what bugs were confirmed, what fixes were shipped, what review findings recurred. Use both: CLAUDE.md for conventions, ForgeDock for institutional memory.
|
|
231
|
+
|
|
232
|
+
**vs. Cursor Memories** — Cursor Memories are scoped to the Cursor IDE's backend and don't survive a Claude Code context reset. ForgeDock annotations live on GitHub — any agent, any tool, any human can read them. When Claude's context window compacts mid-session, a ForgeDock agent queries GitHub and resumes exactly where it left off.
|
|
233
|
+
|
|
234
|
+
**vs. Devin Knowledge Base** — Devin's KB is proprietary, opaque, and vendor-locked. ForgeDock context is plain GitHub issue comments — readable, editable, and queryable with `gh api`. No vendor lock-in, no black box.
|
|
235
|
+
|
|
236
|
+
[Full comparison →](docs/comparison.md)
|
|
237
|
+
|
|
238
|
+
---
|
|
239
|
+
|
|
226
240
|
## Documentation
|
|
227
241
|
|
|
228
242
|
- [Getting Started in 5 Minutes](docs/site/getting-started.md) — install, configure, first pipeline run
|
|
229
243
|
- [How the Knowledge Graph Works](docs/site/how-it-works.md) — FORGE annotations, context relay, compaction resilience
|
|
230
244
|
- [ForgeDock vs. Manual Workflows](docs/site/vs-manual-workflows.md) — structured pipelines vs. ad-hoc prompting
|
|
245
|
+
- [ForgeDock vs. DeepWiki, AGENTS.md, and Cursor Memories](docs/comparison.md) — structural differences between context tools
|
|
231
246
|
- [FORGE Annotation Protocol](docs/site/forge-annotation-protocol.md) — open standard spec for AI context passing
|
|
232
247
|
- [Command Reference](docs/site/command-reference.md) — all 25+ commands with usage and examples
|
|
233
248
|
|
|
@@ -0,0 +1,469 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Stack-aware CI gap detection — audits a project's GitHub Actions workflows against its declared tech stack and files issues for missing config validation checks
|
|
3
|
+
argument-hint: [--repo <prefix>] [--dry-run]
|
|
4
|
+
---
|
|
5
|
+
<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios -->
|
|
6
|
+
<!-- SPDX-License-Identifier: AGPL-3.0-or-later -->
|
|
7
|
+
|
|
8
|
+
# /ci-audit — Stack-Aware CI Gap Detection
|
|
9
|
+
|
|
10
|
+
**Input**: $ARGUMENTS
|
|
11
|
+
|
|
12
|
+
You are the CI hygiene auditor. Given a project's declared tech stack (from `forge.yaml`), determine what config validation tools exist for each stack component and audit whether the project's GitHub Actions workflows include those validation steps. File GitHub issues for any gaps found.
|
|
13
|
+
|
|
14
|
+
This command enforces ForgeDock's separation-of-concerns principle:
|
|
15
|
+
|
|
16
|
+
| Layer | What runs | Tokens | Example |
|
|
17
|
+
|-------|-----------|--------|---------|
|
|
18
|
+
| **CI (automated)** | Deterministic tool validators | Zero | `traefik validate`, `nginx -t`, `kubectl --dry-run` |
|
|
19
|
+
| **ForgeDock (autonomous)** | Reasoning about architecture, logic, security | Yes | "Is this timeout chain coherent?", "Does this wiring match the deploy script?" |
|
|
20
|
+
| **ForgeDock CI hygiene** | Audit that CI has the right checks for the stack | Once | "Project uses Traefik v3 but CI has no `traefik validate` step → file issue" |
|
|
21
|
+
|
|
22
|
+
Config validation runs in CI (zero tokens). ForgeDock's role is to ensure the right CI checks exist — not to run them itself. <!-- Added: forge#1104 -->
|
|
23
|
+
|
|
24
|
+
This command is designed to be run:
|
|
25
|
+
- **Once after project setup** to establish CI hygiene baseline
|
|
26
|
+
- **After stack changes** (adding Traefik, upgrading K8s, adopting Terraform)
|
|
27
|
+
- **As part of `/autopilot`** periodic review cycle
|
|
28
|
+
- **After a production incident** caused by a structurally invalid config
|
|
29
|
+
|
|
30
|
+
This is NOT a PR review — it does not approve or block. It creates issues for each CI gap found.
|
|
31
|
+
|
|
32
|
+
**Agent model policy**: Default `model: "sonnet"`. If Sonnet is rate-limited, fall back to `model: "opus"`.
|
|
33
|
+
**NEVER use plan mode (EnterPlanMode).**
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Config Preamble
|
|
38
|
+
|
|
39
|
+
Before executing any phase, read `forge.yaml` to resolve project references:
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
GH_REPO=$(yq '.project.owner + "/" + .project.repo' forge.yaml 2>/dev/null)
|
|
43
|
+
GH_FLAG="-R ${GH_REPO}"
|
|
44
|
+
REPO_PATH=$(yq '.paths.root' forge.yaml 2>/dev/null || echo ".")
|
|
45
|
+
TECH_STACK=$(yq '.review.tech_stack // ""' forge.yaml 2>/dev/null || echo "")
|
|
46
|
+
TOOL_VERSIONS=$(yq '.review.tool_versions // {}' forge.yaml 2>/dev/null || echo '{}')
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
**Optional `forge.yaml` fields used by this command**:
|
|
50
|
+
|
|
51
|
+
```yaml
|
|
52
|
+
review:
|
|
53
|
+
tech_stack: "Traefik v3, nginx 1.25, Terraform 1.6, Kubernetes 1.28" # Free-form stack description
|
|
54
|
+
tool_versions: # Structured version declarations
|
|
55
|
+
traefik: "3"
|
|
56
|
+
nginx: "1.25"
|
|
57
|
+
terraform: "1.6"
|
|
58
|
+
kubernetes: "1.28"
|
|
59
|
+
docker_compose: "2"
|
|
60
|
+
|
|
61
|
+
# Optional: override which CI workflows to audit (default: all files in .github/workflows/)
|
|
62
|
+
ci_audit:
|
|
63
|
+
workflow_paths:
|
|
64
|
+
- ".github/workflows/ci.yml"
|
|
65
|
+
- ".github/workflows/deploy-production.yml"
|
|
66
|
+
dry_run: false # When true: print findings without creating issues
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
If `forge.yaml` is missing: stop and tell the user to run `npx forgedock init` to generate it.
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## Phase 1: Discover Stack Components
|
|
74
|
+
|
|
75
|
+
### 1A: Parse declared stack
|
|
76
|
+
|
|
77
|
+
Extract stack components from `forge.yaml → review.tech_stack` (free-form) and `forge.yaml → review.tool_versions` (structured).
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
echo "=== Stack from forge.yaml ==="
|
|
81
|
+
echo "TECH_STACK: $TECH_STACK"
|
|
82
|
+
echo "TOOL_VERSIONS: $TOOL_VERSIONS"
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
**Parse `TECH_STACK` (free-form text) for known tool names**:
|
|
86
|
+
|
|
87
|
+
Scan the `TECH_STACK` string for these tool keywords (case-insensitive):
|
|
88
|
+
|
|
89
|
+
| Keyword pattern | Tool | Validator(s) |
|
|
90
|
+
|-----------------|------|-------------|
|
|
91
|
+
| `traefik` | Traefik | `traefik validate`, `traefik healthcheck` |
|
|
92
|
+
| `nginx` | nginx | `nginx -t`, `nginx -T` |
|
|
93
|
+
| `kubernetes\|k8s\|kubectl` | Kubernetes | `kubectl apply --dry-run=client`, `kubeval`, `kustomize build \| kubectl apply --dry-run=client` |
|
|
94
|
+
| `terraform` | Terraform | `terraform validate`, `terraform plan`, `tflint` |
|
|
95
|
+
| `helm` | Helm | `helm lint`, `helm template \| kubectl apply --dry-run=client` |
|
|
96
|
+
| `docker.compose\|docker-compose` | Docker Compose | `docker compose config` |
|
|
97
|
+
| `pydantic\|fastapi` | Python (Pydantic) | `python -c "from app.main import app"` (startup validation) |
|
|
98
|
+
| `ansible` | Ansible | `ansible-playbook --syntax-check`, `ansible-lint` |
|
|
99
|
+
| `pulumi` | Pulumi | `pulumi preview --non-interactive` |
|
|
100
|
+
|
|
101
|
+
**Parse `TOOL_VERSIONS` (structured)** — for each key present, add to the detected components map with its version.
|
|
102
|
+
|
|
103
|
+
**Version-aware validator selection**:
|
|
104
|
+
|
|
105
|
+
The correct validator depends on the tool version. Version mismatches (e.g., using Traefik v2 flags on a v3 install) produce misleading results.
|
|
106
|
+
|
|
107
|
+
| Tool | Version | Preferred validator | Notes |
|
|
108
|
+
|------|---------|---------------------|-------|
|
|
109
|
+
| Traefik | v2.x | `traefik validate --configFile=traefik.yml` | v2 flag syntax |
|
|
110
|
+
| Traefik | v3.x | `traefik validate` (no `--configFile` flag needed in v3) | v3 changed CLI |
|
|
111
|
+
| nginx | any | `nginx -t -c /etc/nginx/nginx.conf` | Standard across versions |
|
|
112
|
+
| Terraform | ≥0.13 | `terraform validate` (in-directory) | Requires `terraform init` first |
|
|
113
|
+
| Terraform | all | `tflint` (linter, catches schema errors `terraform validate` misses) | Complementary |
|
|
114
|
+
| Kubernetes | any | `kubectl apply --dry-run=client -f k8s/` | Client-side, no cluster needed |
|
|
115
|
+
| Helm | ≥3.x | `helm lint ./chart` | v3 only (v2 deprecated) |
|
|
116
|
+
| Docker Compose | v2 | `docker compose config` | v2 CLI (`docker compose`, not `docker-compose`) |
|
|
117
|
+
| Docker Compose | v1 | `docker-compose config` | v1 CLI (legacy) |
|
|
118
|
+
|
|
119
|
+
If tool version is unknown: use the most permissive/universal validator form and note version ambiguity in the filed issue.
|
|
120
|
+
|
|
121
|
+
### 1B: Discover config file locations
|
|
122
|
+
|
|
123
|
+
For each detected tool, check whether config files exist in the repo:
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
REPO_PATH="${REPO_PATH:-.}"
|
|
127
|
+
|
|
128
|
+
# Traefik
|
|
129
|
+
TRAEFIK_CONFIGS=$(find "$REPO_PATH/traefik" -name "*.yml" -o -name "*.yaml" -o -name "*.toml" 2>/dev/null | head -10)
|
|
130
|
+
[ -n "$TRAEFIK_CONFIGS" ] && echo "Traefik configs found: $TRAEFIK_CONFIGS"
|
|
131
|
+
|
|
132
|
+
# nginx
|
|
133
|
+
NGINX_CONFIGS=$(find "$REPO_PATH/infra/nginx" "$REPO_PATH/nginx" -name "*.conf" 2>/dev/null | head -10)
|
|
134
|
+
[ -n "$NGINX_CONFIGS" ] && echo "nginx configs found: $NGINX_CONFIGS"
|
|
135
|
+
|
|
136
|
+
# Kubernetes
|
|
137
|
+
K8S_CONFIGS=$(find "$REPO_PATH/k8s" "$REPO_PATH/kubernetes" "$REPO_PATH/manifests" -name "*.yml" -o -name "*.yaml" 2>/dev/null | head -10)
|
|
138
|
+
[ -n "$K8S_CONFIGS" ] && echo "Kubernetes manifests found: $K8S_CONFIGS"
|
|
139
|
+
|
|
140
|
+
# Terraform
|
|
141
|
+
TF_CONFIGS=$(find "$REPO_PATH/terraform" "$REPO_PATH/infra/terraform" -name "*.tf" 2>/dev/null | head -10)
|
|
142
|
+
[ -n "$TF_CONFIGS" ] && echo "Terraform configs found: $TF_CONFIGS"
|
|
143
|
+
|
|
144
|
+
# Helm charts
|
|
145
|
+
HELM_CHARTS=$(find "$REPO_PATH" -name "Chart.yaml" 2>/dev/null | head -10)
|
|
146
|
+
[ -n "$HELM_CHARTS" ] && echo "Helm charts found: $HELM_CHARTS"
|
|
147
|
+
|
|
148
|
+
# Docker Compose
|
|
149
|
+
COMPOSE_FILES=$(find "$REPO_PATH" -maxdepth 2 -name "docker-compose*.yml" 2>/dev/null | head -10)
|
|
150
|
+
[ -n "$COMPOSE_FILES" ] && echo "Docker Compose files found: $COMPOSE_FILES"
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
**Skip any tool whose config files are NOT found in the repo** — a declared stack component with no config files has no CI validation gap to audit.
|
|
154
|
+
|
|
155
|
+
---
|
|
156
|
+
|
|
157
|
+
## Phase 2: Audit GitHub Actions Workflows
|
|
158
|
+
|
|
159
|
+
### 2A: Read all workflow files
|
|
160
|
+
|
|
161
|
+
```bash
|
|
162
|
+
WORKFLOW_DIR="$REPO_PATH/.github/workflows"
|
|
163
|
+
WORKFLOW_FILES=$(ls "$WORKFLOW_DIR"/*.yml "$WORKFLOW_DIR"/*.yaml 2>/dev/null | head -30)
|
|
164
|
+
|
|
165
|
+
if [ -z "$WORKFLOW_FILES" ]; then
|
|
166
|
+
echo "AUDIT GAP: No GitHub Actions workflows found in $WORKFLOW_DIR"
|
|
167
|
+
echo "A project without CI has no automated validation at all — see Phase 3 for issue creation."
|
|
168
|
+
fi
|
|
169
|
+
|
|
170
|
+
echo "=== Workflows to audit ==="
|
|
171
|
+
echo "$WORKFLOW_FILES"
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
**Override with `forge.yaml → ci_audit.workflow_paths`** if that section is present:
|
|
175
|
+
```bash
|
|
176
|
+
CUSTOM_PATHS=$(yq '.ci_audit.workflow_paths // []' forge.yaml 2>/dev/null | yq '.[]' 2>/dev/null || echo "")
|
|
177
|
+
[ -n "$CUSTOM_PATHS" ] && WORKFLOW_FILES="$CUSTOM_PATHS"
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
### 2B: Check for each tool's validator
|
|
181
|
+
|
|
182
|
+
For each tool detected in Phase 1 with config files present, search all workflow files for a validation step:
|
|
183
|
+
|
|
184
|
+
```bash
|
|
185
|
+
# Build a combined search corpus from all workflow files
|
|
186
|
+
WORKFLOW_CORPUS=$(cat $WORKFLOW_FILES 2>/dev/null)
|
|
187
|
+
|
|
188
|
+
declare -A TOOL_VALIDATOR_FOUND
|
|
189
|
+
declare -A TOOL_VALIDATOR_PATTERN
|
|
190
|
+
|
|
191
|
+
# Traefik
|
|
192
|
+
TOOL_VALIDATOR_PATTERN[traefik]="traefik.*validate|traefik.*healthcheck|traefik.*check"
|
|
193
|
+
|
|
194
|
+
# nginx
|
|
195
|
+
TOOL_VALIDATOR_PATTERN[nginx]="nginx.*-t|nginx.*-T|nginx.*configtest|nginx.*test"
|
|
196
|
+
|
|
197
|
+
# Kubernetes
|
|
198
|
+
TOOL_VALIDATOR_PATTERN[kubernetes]="kubectl.*--dry-run|kubeval|kustomize.*\|.*kubectl|kubectl.*validate"
|
|
199
|
+
|
|
200
|
+
# Terraform
|
|
201
|
+
TOOL_VALIDATOR_PATTERN[terraform]="terraform.*validate|terraform.*plan|tflint"
|
|
202
|
+
|
|
203
|
+
# Helm
|
|
204
|
+
TOOL_VALIDATOR_PATTERN[helm]="helm.*lint|helm.*template"
|
|
205
|
+
|
|
206
|
+
# Docker Compose
|
|
207
|
+
TOOL_VALIDATOR_PATTERN[docker_compose]="docker.*compose.*config|docker-compose.*config"
|
|
208
|
+
|
|
209
|
+
for tool in "${!TOOL_VALIDATOR_PATTERN[@]}"; do
|
|
210
|
+
PATTERN="${TOOL_VALIDATOR_PATTERN[$tool]}"
|
|
211
|
+
if echo "$WORKFLOW_CORPUS" | grep -qiE "$PATTERN"; then
|
|
212
|
+
TOOL_VALIDATOR_FOUND[$tool]="YES"
|
|
213
|
+
FOUND_IN=$(grep -liE "$PATTERN" $WORKFLOW_FILES 2>/dev/null | head -3 | tr '\n' ', ')
|
|
214
|
+
echo "OK: $tool validator found in: $FOUND_IN"
|
|
215
|
+
else
|
|
216
|
+
TOOL_VALIDATOR_FOUND[$tool]="NO"
|
|
217
|
+
echo "GAP: No $tool validation step found in any CI workflow"
|
|
218
|
+
fi
|
|
219
|
+
done
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
### 2C: Check validator placement
|
|
223
|
+
|
|
224
|
+
A validation step that runs only in a deployment workflow (not in the CI workflow that runs on every PR) provides partial coverage — config errors reach `main` undetected if the deploy-only validator only runs at deploy time.
|
|
225
|
+
|
|
226
|
+
```bash
|
|
227
|
+
CI_WORKFLOW=$(ls "$WORKFLOW_DIR/ci.yml" "$WORKFLOW_DIR/test.yml" "$WORKFLOW_DIR/build.yml" 2>/dev/null | head -1)
|
|
228
|
+
|
|
229
|
+
if [ -n "$CI_WORKFLOW" ]; then
|
|
230
|
+
echo "=== Checking validator placement (CI workflow: $CI_WORKFLOW) ==="
|
|
231
|
+
for tool in "${!TOOL_VALIDATOR_FOUND[@]}"; do
|
|
232
|
+
if [ "${TOOL_VALIDATOR_FOUND[$tool]}" = "YES" ]; then
|
|
233
|
+
PATTERN="${TOOL_VALIDATOR_PATTERN[$tool]}"
|
|
234
|
+
if grep -qiE "$PATTERN" "$CI_WORKFLOW" 2>/dev/null; then
|
|
235
|
+
echo "OK: $tool validator is in the CI workflow (runs on every PR)"
|
|
236
|
+
else
|
|
237
|
+
echo "PLACEMENT GAP: $tool validator exists but is NOT in $CI_WORKFLOW — it may only run at deploy time, missing PR-time validation"
|
|
238
|
+
fi
|
|
239
|
+
fi
|
|
240
|
+
done
|
|
241
|
+
fi
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
---
|
|
245
|
+
|
|
246
|
+
## Phase 3: File Issues for Gaps
|
|
247
|
+
|
|
248
|
+
For each gap detected (tool present in repo but no CI validation step found), file a GitHub issue.
|
|
249
|
+
|
|
250
|
+
**Skip issue creation if `--dry-run` flag is passed** (or `forge.yaml → ci_audit.dry_run: true`):
|
|
251
|
+
```bash
|
|
252
|
+
DRY_RUN=$(yq '.ci_audit.dry_run // "false"' forge.yaml 2>/dev/null || echo "false")
|
|
253
|
+
[ "$1" = "--dry-run" ] || [ "$2" = "--dry-run" ] && DRY_RUN="true"
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
### 3A: Determine validator recommendation
|
|
257
|
+
|
|
258
|
+
For each gap, compose a concrete, version-aware recommendation:
|
|
259
|
+
|
|
260
|
+
```bash
|
|
261
|
+
get_validator_recommendation() {
|
|
262
|
+
local tool="$1"
|
|
263
|
+
local version="$2" # may be empty if unknown
|
|
264
|
+
|
|
265
|
+
case "$tool" in
|
|
266
|
+
traefik)
|
|
267
|
+
if [ "${version%%.*}" = "3" ] || echo "$version" | grep -q "^v3"; then
|
|
268
|
+
echo "Add a CI step that runs: \`docker run --rm -v \$(pwd)/traefik:/etc/traefik traefik:v${version:-3} validate\`"
|
|
269
|
+
echo "Traefik v3 schema differs from v2 — the validator must use the same version as production."
|
|
270
|
+
else
|
|
271
|
+
echo "Add a CI step that runs: \`docker run --rm -v \$(pwd)/traefik:/etc/traefik traefik:v${version:-2} validate --configFile=/etc/traefik/traefik.yml\`"
|
|
272
|
+
fi
|
|
273
|
+
;;
|
|
274
|
+
nginx)
|
|
275
|
+
echo "Add a CI step that runs: \`docker run --rm -v \$(pwd)/infra/nginx:/etc/nginx nginx:${version:-alpine} nginx -t\`"
|
|
276
|
+
;;
|
|
277
|
+
kubernetes)
|
|
278
|
+
echo "Add a CI step that runs: \`kubectl apply --dry-run=client -f k8s/\` (requires \`kubectl\` in CI environment, no cluster needed for client-side validation)"
|
|
279
|
+
;;
|
|
280
|
+
terraform)
|
|
281
|
+
echo "Add CI steps that run: (1) \`terraform init -backend=false\` then (2) \`terraform validate\`. Optionally add \`tflint\` for schema linting beyond what terraform validate catches."
|
|
282
|
+
;;
|
|
283
|
+
helm)
|
|
284
|
+
echo "Add a CI step that runs: \`helm lint ./chart\` (or the path to your chart directory)"
|
|
285
|
+
;;
|
|
286
|
+
docker_compose)
|
|
287
|
+
if [ "${version%%.*}" = "2" ]; then
|
|
288
|
+
echo "Add a CI step that runs: \`docker compose config\` (validates compose file structure and interpolation)"
|
|
289
|
+
else
|
|
290
|
+
echo "Add a CI step that runs: \`docker-compose config\` (validates compose file structure and interpolation)"
|
|
291
|
+
fi
|
|
292
|
+
;;
|
|
293
|
+
esac
|
|
294
|
+
}
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
### 3B: Create issues for gaps
|
|
298
|
+
|
|
299
|
+
For each tool with `TOOL_VALIDATOR_FOUND[$tool] = "NO"`:
|
|
300
|
+
|
|
301
|
+
```bash
|
|
302
|
+
for tool in "${!TOOL_VALIDATOR_FOUND[@]}"; do
|
|
303
|
+
if [ "${TOOL_VALIDATOR_FOUND[$tool]}" = "NO" ]; then
|
|
304
|
+
TOOL_VERSION=$(yq ".review.tool_versions.${tool} // \"\"" forge.yaml 2>/dev/null || echo "")
|
|
305
|
+
RECOMMENDATION=$(get_validator_recommendation "$tool" "$TOOL_VERSION")
|
|
306
|
+
VERSION_NOTE=""
|
|
307
|
+
[ -n "$TOOL_VERSION" ] && VERSION_NOTE="**Declared version**: \`$tool $TOOL_VERSION\`" || VERSION_NOTE="**Version**: Unknown — declare in \`forge.yaml → review.tool_versions.$tool\` for version-aware validation"
|
|
308
|
+
|
|
309
|
+
ISSUE_BODY="## Problem
|
|
310
|
+
|
|
311
|
+
The project uses \`$tool\` (config files present in repo) but the CI pipeline has no structural validation step for \`$tool\` config files. Structural config errors (wrong nesting, missing required keys, version-incompatible syntax) are therefore not caught before merge or deploy.
|
|
312
|
+
|
|
313
|
+
**Upstream incident**: This CI gap was identified via \`/ci-audit\` following incident analysis showing that review agents verify logical correctness (coherent values, naming consistency, cross-file wiring) but cannot substitute for deterministic structural validators. <!-- Added: forge#1104 -->
|
|
314
|
+
|
|
315
|
+
## Root Cause
|
|
316
|
+
|
|
317
|
+
**Failure pattern**: The \`$tool\` validator has not been added to CI. Config changes are reviewed for logical correctness by the INFRA review agent but structural correctness (schema conformance) is not deterministically checked before merge.
|
|
318
|
+
|
|
319
|
+
**Separation of concerns**:
|
|
320
|
+
- ForgeDock review agents: verify logical correctness (architecture, wiring, security)
|
|
321
|
+
- CI validators: verify structural correctness (schema conformance, tool version compatibility)
|
|
322
|
+
|
|
323
|
+
Both layers are required. This issue tracks the CI layer gap.
|
|
324
|
+
|
|
325
|
+
## Affected Files
|
|
326
|
+
|
|
327
|
+
Files to be identified during investigation:
|
|
328
|
+
1. \`.github/workflows/ci.yml\` (or equivalent) — add validation step
|
|
329
|
+
2. \`$tool/\` config directory — files to be validated
|
|
330
|
+
|
|
331
|
+
## Expected Behavior
|
|
332
|
+
|
|
333
|
+
Every PR that changes \`$tool\` config files triggers a CI step that runs the \`$tool\` validator. Structural errors (wrong YAML nesting, missing required keys, version-incompatible syntax) are caught before merge — not after deploy.
|
|
334
|
+
|
|
335
|
+
## Acceptance Criteria
|
|
336
|
+
|
|
337
|
+
- [ ] A CI step runs \`$tool\` structural validation on every PR that changes \`$tool\` config files
|
|
338
|
+
- [ ] The CI step uses the same \`$tool\` version as production (version-pinned)
|
|
339
|
+
- [ ] The CI step runs in the PR-time workflow (not only at deploy time)
|
|
340
|
+
- [ ] Validation failure blocks merge
|
|
341
|
+
- [ ] No regression in existing CI workflows
|
|
342
|
+
|
|
343
|
+
## Recommendation
|
|
344
|
+
|
|
345
|
+
${RECOMMENDATION}
|
|
346
|
+
|
|
347
|
+
${VERSION_NOTE}
|
|
348
|
+
|
|
349
|
+
**Where to add**: Add the validation step to the CI workflow that runs on every PR (\`ci.yml\` or equivalent), not only the deploy workflow. A validator that only runs at deploy time allows structural errors to merge to \`staging\` undetected.
|
|
350
|
+
|
|
351
|
+
**Implementation**: File this issue → \`/work-on\` pipeline → builder adds the CI step → review verifies version pinning and trigger conditions → merge.
|
|
352
|
+
|
|
353
|
+
## Severity
|
|
354
|
+
|
|
355
|
+
**P2** — Structural config errors are silently ignored by the tool at runtime. A project without a \`$tool\` validator in CI will ship broken configs whenever a config refactor changes YAML nesting. The failure mode is silent and only surfaces at deploy time (or not at all, if the tool silently ignores the misconfiguration)."
|
|
356
|
+
|
|
357
|
+
if [ "$DRY_RUN" = "true" ]; then
|
|
358
|
+
echo "=== DRY RUN: Would create issue for $tool CI gap ==="
|
|
359
|
+
echo "Title: ci($tool): add structural config validation step to CI pipeline"
|
|
360
|
+
echo "Body:"
|
|
361
|
+
echo "$ISSUE_BODY"
|
|
362
|
+
echo "Labels: bug,P2,audit-finding"
|
|
363
|
+
else
|
|
364
|
+
CREATED_ISSUE=$(gh issue create $GH_FLAG \
|
|
365
|
+
--title "ci($tool): add structural config validation step to CI pipeline" \
|
|
366
|
+
--body "$ISSUE_BODY" \
|
|
367
|
+
--label "bug,P2,audit-finding" 2>&1)
|
|
368
|
+
echo "Created issue for $tool gap: $CREATED_ISSUE"
|
|
369
|
+
fi
|
|
370
|
+
fi
|
|
371
|
+
done
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
### 3C: Handle placement gaps
|
|
375
|
+
|
|
376
|
+
For each tool whose validator exists but is only in the deploy workflow (not in the CI workflow):
|
|
377
|
+
|
|
378
|
+
```bash
|
|
379
|
+
# (Populated in Phase 2C)
|
|
380
|
+
for tool in "${!PLACEMENT_GAPS[@]}"; do
|
|
381
|
+
TOOL_VERSION=$(yq ".review.tool_versions.${tool} // \"\"" forge.yaml 2>/dev/null || echo "")
|
|
382
|
+
|
|
383
|
+
ISSUE_BODY="## Problem
|
|
384
|
+
|
|
385
|
+
The project has a \`$tool\` validation step in CI, but it only runs in the deployment workflow — not in the PR-time CI workflow. This means structural config errors in \`$tool\` configs can merge to \`staging\` undetected and only surface during deployment.
|
|
386
|
+
|
|
387
|
+
## Root Cause
|
|
388
|
+
|
|
389
|
+
**Failure pattern**: Validator placement is deploy-time only. PRs that introduce structural config errors pass all checks and merge normally. The error is only caught when the deploy pipeline runs.
|
|
390
|
+
|
|
391
|
+
## Acceptance Criteria
|
|
392
|
+
|
|
393
|
+
- [ ] The \`$tool\` validation step runs in the PR-time CI workflow (\`ci.yml\` or equivalent)
|
|
394
|
+
- [ ] Validation failure blocks merge (not just alerts)
|
|
395
|
+
- [ ] No regression in the existing deploy-time validation
|
|
396
|
+
|
|
397
|
+
## Severity
|
|
398
|
+
|
|
399
|
+
**P3** — Partial coverage. Structural errors are caught at deploy time but not at PR time, creating a window where broken configs exist on \`staging\` until a deploy is attempted."
|
|
400
|
+
|
|
401
|
+
if [ "$DRY_RUN" = "true" ]; then
|
|
402
|
+
echo "=== DRY RUN: Would create placement issue for $tool ==="
|
|
403
|
+
else
|
|
404
|
+
gh issue create $GH_FLAG \
|
|
405
|
+
--title "ci($tool): move config validation to PR-time CI workflow" \
|
|
406
|
+
--body "$ISSUE_BODY" \
|
|
407
|
+
--label "bug,P3,audit-finding" 2>&1
|
|
408
|
+
fi
|
|
409
|
+
done
|
|
410
|
+
```
|
|
411
|
+
|
|
412
|
+
---
|
|
413
|
+
|
|
414
|
+
## Phase 4: Post Audit Summary
|
|
415
|
+
|
|
416
|
+
```bash
|
|
417
|
+
gh issue comment $(gh issue list $GH_FLAG --search "ci-audit" --limit 1 --json number --jq '.[0].number // ""' 2>/dev/null || echo "") $GH_FLAG --body "## CI Audit Summary
|
|
418
|
+
|
|
419
|
+
**Audit date**: $(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
|
420
|
+
**Stack audited**: ${TECH_STACK:-"(no tech_stack declared in forge.yaml)"}
|
|
421
|
+
|
|
422
|
+
### Results
|
|
423
|
+
|
|
424
|
+
| Tool | Config Files Found | CI Validator Found | Placement | Issues Filed |
|
|
425
|
+
|------|--------------------|-------------------|-----------|-------------|
|
|
426
|
+
$(for tool in "${!TOOL_VALIDATOR_FOUND[@]}"; do
|
|
427
|
+
CONFIG_FOUND=$([ -n "${TOOL_CONFIG_FOUND[$tool]}" ] && echo "Yes" || echo "No")
|
|
428
|
+
VALIDATOR="${TOOL_VALIDATOR_FOUND[$tool]:-NO}"
|
|
429
|
+
PLACEMENT="${TOOL_PLACEMENT[$tool]:-"N/A"}"
|
|
430
|
+
echo "| $tool | $CONFIG_FOUND | $VALIDATOR | $PLACEMENT | (see above) |"
|
|
431
|
+
done)
|
|
432
|
+
|
|
433
|
+
### Actions Required
|
|
434
|
+
|
|
435
|
+
$([ -n "$(for t in "${!TOOL_VALIDATOR_FOUND[@]}"; do [ "${TOOL_VALIDATOR_FOUND[$t]}" = "NO" ] && echo "$t"; done)" ] && \
|
|
436
|
+
echo "Issues filed for: $(for t in "${!TOOL_VALIDATOR_FOUND[@]}"; do [ "${TOOL_VALIDATOR_FOUND[$t]}" = "NO" ] && echo "$t"; done | tr '\n' ' ')" || \
|
|
437
|
+
echo "No CI gaps found — all detected stack components have CI validators.")
|
|
438
|
+
|
|
439
|
+
### Next Steps
|
|
440
|
+
|
|
441
|
+
1. Run \`/work-on\` on each filed issue to implement the CI validation step
|
|
442
|
+
2. Re-run \`/ci-audit\` after implementation to verify gaps are closed
|
|
443
|
+
3. Add \`forge.yaml → review.tool_versions\` entries for any tools whose version was marked Unknown
|
|
444
|
+
" 2>/dev/null || echo "Audit complete. Summary printed above (no issue to comment on in dry-run mode)."
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
Print the summary to stdout regardless:
|
|
448
|
+
|
|
449
|
+
```bash
|
|
450
|
+
echo ""
|
|
451
|
+
echo "=== /ci-audit Complete ==="
|
|
452
|
+
echo "Stack: ${TECH_STACK:-"(no tech_stack declared)"}"
|
|
453
|
+
echo "Workflows audited: $(echo $WORKFLOW_FILES | tr ' ' '\n' | wc -l | tr -d ' ')"
|
|
454
|
+
echo "Tools with config files: $(echo "${!TOOL_CONFIG_FOUND[@]}" | wc -w | tr -d ' ')"
|
|
455
|
+
echo "CI gaps found: $(for t in "${!TOOL_VALIDATOR_FOUND[@]}"; do [ "${TOOL_VALIDATOR_FOUND[$t]}" = "NO" ] && echo "$t"; done | wc -w | tr -d ' ')"
|
|
456
|
+
echo ""
|
|
457
|
+
echo "Issues filed:"
|
|
458
|
+
echo "${ISSUES_FILED:-" (none — dry-run or no gaps)"}"
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
---
|
|
462
|
+
|
|
463
|
+
## Error Handling
|
|
464
|
+
|
|
465
|
+
- **`forge.yaml` missing**: Stop and tell the user to run `npx forgedock init`
|
|
466
|
+
- **No stack declared** (`TECH_STACK` empty and `TOOL_VERSIONS` empty): Print advisory — "No tech stack declared in forge.yaml. Add `review.tech_stack` or `review.tool_versions` to enable stack-aware CI auditing." Do not file issues.
|
|
467
|
+
- **No config files found for a declared tool**: Skip that tool's CI audit — presence in `TECH_STACK` without actual config files may mean the project is in the process of adopting the tool, not that CI is missing a validator.
|
|
468
|
+
- **No GitHub Actions workflows found**: File a single high-severity issue: "Project has no CI — all stack components lack automated validation."
|
|
469
|
+
- **`gh` auth failure**: Print `gh auth status` output and stop.
|