forgedock 1.0.20 → 1.0.22
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +146 -77
- package/bin/demo.mjs +386 -0
- package/bin/forgedock.mjs +357 -30
- package/bin/init-detect.mjs +95 -0
- package/bin/runner.mjs +780 -0
- package/commands/ci-audit.md +469 -0
- package/commands/cleanup.md +85 -39
- package/commands/deploy-info.md +6 -2
- package/commands/forge-stats.md +1 -1
- package/commands/optimize.md +1 -1
- package/commands/orchestrate.md +684 -33
- package/commands/quality-gate.md +166 -46
- package/commands/review-pr-agents.md +106 -34
- package/commands/review-pr.md +33 -2
- package/commands/scope.md +11 -3
- package/commands/security-audit.md +24 -16
- package/commands/sync-ecosystem.md +2 -2
- package/commands/work-on/build/architect.md +47 -3
- package/commands/work-on/build/context.md +36 -5
- package/commands/work-on/build/validate.md +16 -7
- package/commands/work-on/close.md +129 -1
- package/commands/work-on/investigate.md +24 -1
- package/commands/work-on.md +226 -8
- package/docs/comparison.md +135 -0
- package/docs/design/developer-objections-ai-context-tools.md +244 -0
- package/docs/design/messaging-angles-research.md +179 -0
- package/docs/design/show-hn-draft.md +86 -0
- package/docs/site/annotations-explained.md +55 -0
- package/docs/site/command-learning-path.md +104 -0
- package/docs/site/command-reference.md +3 -0
- package/docs/site/getting-started.md +36 -14
- package/docs/site/troubleshooting.md +208 -0
- package/docs/spec/forge-protocol-v1.md +768 -0
- package/examples/forgedock-demo/README.md +113 -0
- package/examples/forgedock-demo/bootstrap.sh +125 -0
- package/examples/forgedock-demo/issues/01-bug-delete-missing-auth.md +37 -0
- package/examples/forgedock-demo/issues/02-feature-filter-notes.md +37 -0
- package/examples/forgedock-demo/issues/03-refactor-router-table.md +33 -0
- package/examples/forgedock-demo/issues/04-perf-index-lookup.md +35 -0
- package/examples/forgedock-demo/issues/05-docs-api-reference.md +39 -0
- package/examples/forgedock-demo/labels.json +21 -0
- package/examples/forgedock-demo/package.json +16 -0
- package/examples/forgedock-demo/scripts/smoke.js +61 -0
- package/examples/forgedock-demo/src/auth.js +28 -0
- package/examples/forgedock-demo/src/db.js +66 -0
- package/examples/forgedock-demo/src/routes/notes.js +64 -0
- package/examples/forgedock-demo/src/server.js +79 -0
- package/package.json +6 -2
- package/templates/forge.yaml.minimal +44 -0
package/commands/quality-gate.md
CHANGED
|
@@ -48,6 +48,7 @@ This quality gate uses **domain detection** to adapt to your project's actual te
|
|
|
48
48
|
| INFRA | Dockerfiles or entrypoints with runtime UID changes |
|
|
49
49
|
| ROUTER_BUG | Router Python files where gate conditions are narrowed |
|
|
50
50
|
| FORGE_GRAPH | `commands/*.md` specs or `scripts/*` files change (ForgeDock self-consistency) |
|
|
51
|
+
| CONFIG_SCHEMA | Config files for external tools (`traefik/`, `infra/nginx/`, `k8s/`, `terraform/`, `*.conf`, `*.toml` in infra paths, or `docker-compose*.yml` with service definition changes) |
|
|
51
52
|
|
|
52
53
|
**Stack-agnostic coverage**: The example domains above reflect common patterns caught in production; they are not requirements. A Go or Ruby project benefits from SECURITY checks. A Node.js project benefits from FRONTEND, PROXY, and DEPLOY checks. Any project benefits from DATABASE and WORKFLOW checks when those file types are present. The stack-specific examples (Python routers, SOPS secrets chain, FastAPI layouts, appleboy SSH deploys) are illustrative — the domain detection system applies the applicable subset to any codebase.
|
|
53
54
|
|
|
@@ -100,6 +101,7 @@ Before running checks, classify the changed files into domains. This avoids runn
|
|
|
100
101
|
| `Dockerfile*` or `entrypoint*.sh` with added/changed `USER`, `su-exec`, `gosu`, or `setuid` | INFRA |
|
|
101
102
|
| `*.py` in `routers/` where the diff removes a line containing an or-gate condition (lines starting with `-` containing `if.*or`) | ROUTER_BUG |
|
|
102
103
|
| `commands/*.md` or `scripts/*` files (ForgeDock repo only — dogfoods its own spec graph) | FORGE_GRAPH |
|
|
104
|
+
| Files under `traefik/`, `infra/nginx/`, `k8s/`, `terraform/`, or files matching `*.conf`, `*.toml` in infra paths, or `docker-compose*.yml` with service definition changes | CONFIG_SCHEMA |
|
|
103
105
|
|
|
104
106
|
**Apply the classification:**
|
|
105
107
|
|
|
@@ -107,7 +109,12 @@ Before running checks, classify the changed files into domains. This avoids runn
|
|
|
107
109
|
# Initialize domain flags
|
|
108
110
|
DOMAINS=""
|
|
109
111
|
|
|
110
|
-
|
|
112
|
+
# {CHANGED_FILES} is a space-separated argument (same contract as {AFFECTED_FILES}
|
|
113
|
+
# in architect.md) — split explicitly on IFS=' ' into an array instead of relying
|
|
114
|
+
# on a bare `for f in {CHANGED_FILES}`, which word-splits on the shell's default
|
|
115
|
+
# IFS (space, tab, AND newline) and would corrupt any path containing a space.
|
|
116
|
+
IFS=' ' read -ra CHANGED_FILES_ARR <<< "{CHANGED_FILES}"
|
|
117
|
+
for f in "${CHANGED_FILES_ARR[@]}"; do
|
|
111
118
|
case "$f" in
|
|
112
119
|
*.sql|*migrations/*) DOMAINS="$DOMAINS DATABASE" ;;
|
|
113
120
|
esac
|
|
@@ -142,30 +149,49 @@ grep -lE "asyncio\.shield|asyncio\.wait_for|Task\.cancel" {CHANGED_FILES} 2>/dev
|
|
|
142
149
|
grep -lE "^\w+ = (\{\}|\[\]|set\(\)|Lock\(\)|defaultdict|Counter)" {CHANGED_FILES} 2>/dev/null | grep -qE '\.py$' && DOMAINS="$DOMAINS STATE"
|
|
143
150
|
|
|
144
151
|
# Check for capacity constants in worker/infra/browser Python files
|
|
145
|
-
|
|
152
|
+
while IFS= read -r f; do
|
|
153
|
+
[ -z "$f" ] && continue
|
|
146
154
|
git diff HEAD -- "$f" 2>/dev/null | grep -qE '^\+[A-Za-z_]\w*(MB|SIZE|MAX|LIMIT|THRESHOLD|TIMEOUT)\w*\s*=' && DOMAINS="$DOMAINS CAPACITY" && break
|
|
147
|
-
done
|
|
155
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'services/worker/|infra/|browser/')
|
|
148
156
|
|
|
149
157
|
# Check for env var usage in changed files
|
|
150
158
|
grep -lE "os\.getenv|process\.env" {CHANGED_FILES} 2>/dev/null && DOMAINS="$DOMAINS DEPLOY"
|
|
151
159
|
|
|
152
160
|
# Check for new app.* imports added outside try: blocks in Python files
|
|
153
|
-
|
|
161
|
+
while IFS= read -r f; do
|
|
162
|
+
[ -z "$f" ] && continue
|
|
154
163
|
NEW_IMPORTS=$(git diff HEAD -- "$f" 2>/dev/null | grep -E '^\+\s*(from app\.|import app\.)' | grep -v '^+++')
|
|
155
164
|
if [ -n "$NEW_IMPORTS" ]; then
|
|
156
165
|
DOMAINS="$DOMAINS IMPORT_RESOLUTION"
|
|
157
166
|
break
|
|
158
167
|
fi
|
|
159
|
-
done
|
|
168
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$')
|
|
160
169
|
|
|
161
170
|
# Check for runtime UID changes in Dockerfiles or entrypoint scripts
|
|
162
|
-
|
|
171
|
+
while IFS= read -r f; do
|
|
172
|
+
[ -z "$f" ] && continue
|
|
163
173
|
git diff HEAD -- "$f" 2>/dev/null | grep -E '^\+.*(USER\s+[^0]|su-exec|gosu|setuid)' | grep -v '^+++' | grep -q . && DOMAINS="$DOMAINS INFRA" && break
|
|
164
|
-
done
|
|
174
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -iE 'Dockerfile|entrypoint.*\.sh')
|
|
165
175
|
|
|
166
176
|
# Check for gate condition narrowing in router Python files (condition removal = fix may be incomplete across siblings)
|
|
167
|
-
|
|
177
|
+
while IFS= read -r f; do
|
|
178
|
+
[ -z "$f" ] && continue
|
|
168
179
|
git diff HEAD -- "$f" 2>/dev/null | grep -E '^-.*\bif\b.*\bor\b' | grep -v '^---' | grep -q . && DOMAINS="$DOMAINS ROUTER_BUG" && break
|
|
180
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'router')
|
|
181
|
+
|
|
182
|
+
# Check for external tool config files (structural schema validation advisory)
|
|
183
|
+
# {CHANGED_FILES} is space-separated (see array-split note above) — reuse the
|
|
184
|
+
# same CHANGED_FILES_ARR built above instead of re-splitting.
|
|
185
|
+
for f in "${CHANGED_FILES_ARR[@]}"; do
|
|
186
|
+
case "$f" in
|
|
187
|
+
traefik/*|infra/nginx/*|k8s/*|terraform/*)
|
|
188
|
+
DOMAINS="$DOMAINS CONFIG_SCHEMA" && break ;;
|
|
189
|
+
*.conf|*.toml)
|
|
190
|
+
echo "$f" | grep -qE 'infra/|traefik/|nginx/|k8s/|terraform/' && DOMAINS="$DOMAINS CONFIG_SCHEMA" && break ;;
|
|
191
|
+
docker-compose*.yml)
|
|
192
|
+
# Trigger only if service definitions changed (not just env vars or labels)
|
|
193
|
+
git diff HEAD -- "$f" 2>/dev/null | grep -E '^\+\s+(image:|command:|entrypoint:)' | grep -q . && DOMAINS="$DOMAINS CONFIG_SCHEMA" && break ;;
|
|
194
|
+
esac
|
|
169
195
|
done
|
|
170
196
|
|
|
171
197
|
# SECURITY is always included for any code file
|
|
@@ -190,7 +216,7 @@ Run ONLY the checks whose domain was identified in Step 1.5. Skip checks for dom
|
|
|
190
216
|
- **2D (Database)**: Run if `DATABASE` in DOMAINS
|
|
191
217
|
- **2E (Frontend lifecycle)**: Run if `FRONTEND` in DOMAINS
|
|
192
218
|
- **2F (Frontend proxy)**: Run if `PROXY` in DOMAINS
|
|
193
|
-
- **2G (Cross-service integration)**: Run if `SHELL` in DOMAINS
|
|
219
|
+
- **2G (Cross-service integration)**: Run if `SHELL` or `FORGE_GRAPH` in DOMAINS
|
|
194
220
|
- **2H (Asyncio cancellation)**: Run if `CONCURRENCY` in DOMAINS
|
|
195
221
|
- **2I (State completeness)**: Run if `STATE` in DOMAINS
|
|
196
222
|
- **2J (String literal consistency)**: Run if `STRING_SETS` in DOMAINS
|
|
@@ -199,6 +225,7 @@ Run ONLY the checks whose domain was identified in Step 1.5. Skip checks for dom
|
|
|
199
225
|
- **2M (Import resolution)**: Run if `IMPORT_RESOLUTION` in DOMAINS
|
|
200
226
|
- **2N (Runtime UID × filesystem write)**: Run if `INFRA` in DOMAINS
|
|
201
227
|
- **2O (Residual pattern check)**: Run if `ROUTER_BUG` in DOMAINS
|
|
228
|
+
- **2P (External tool config schema)**: Run if `CONFIG_SCHEMA` in DOMAINS
|
|
202
229
|
|
|
203
230
|
### 2A: Security (ALL files)
|
|
204
231
|
|
|
@@ -212,7 +239,11 @@ For every changed file, check:
|
|
|
212
239
|
|
|
213
240
|
```bash
|
|
214
241
|
# Quick automated checks
|
|
215
|
-
|
|
242
|
+
# {CHANGED_FILES} is a space-separated argument — split explicitly on IFS=' '
|
|
243
|
+
# into an array instead of a bare `for f in {CHANGED_FILES}` (see Step 1.5 for
|
|
244
|
+
# the full rationale; this is a separate bash block so the array is rebuilt here).
|
|
245
|
+
IFS=' ' read -ra CHANGED_FILES_ARR <<< "{CHANGED_FILES}"
|
|
246
|
+
for f in "${CHANGED_FILES_ARR[@]}"; do
|
|
216
247
|
# f-string SQL
|
|
217
248
|
grep -nE "f['\"].*SELECT|f['\"].*INSERT|f['\"].*UPDATE|f['\"].*DELETE|f['\"].*WHERE" "$f" 2>/dev/null && echo "SEC: f-string SQL in $f"
|
|
218
249
|
# Hardcoded secrets
|
|
@@ -240,10 +271,11 @@ For new/modified endpoints:
|
|
|
240
271
|
3. Does the endpoint check resource ownership (`user_id`) before returning data?
|
|
241
272
|
|
|
242
273
|
```bash
|
|
243
|
-
|
|
274
|
+
while IFS= read -r f; do
|
|
275
|
+
[ -z "$f" ] && continue
|
|
244
276
|
grep -nE "CurrentUser|SessionUser|Depends\(get_" "$f" 2>/dev/null
|
|
245
277
|
grep -nE "@router\.(get|post|put|delete)" "$f" 2>/dev/null
|
|
246
|
-
done
|
|
278
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E 'router|route')
|
|
247
279
|
```
|
|
248
280
|
|
|
249
281
|
### 2C: Deploy chain (when new env vars are introduced)
|
|
@@ -270,11 +302,12 @@ For each `.sql` file:
|
|
|
270
302
|
5. **Migration number collision**: Check against origin branch
|
|
271
303
|
|
|
272
304
|
```bash
|
|
273
|
-
|
|
305
|
+
while IFS= read -r f; do
|
|
306
|
+
[ -z "$f" ] && continue
|
|
274
307
|
grep -nE "ADD COLUMN.*NOT NULL" "$f" | grep -v "DEFAULT" && echo "DB: NOT NULL without DEFAULT in $f"
|
|
275
308
|
grep -nE "DROP (TABLE|COLUMN|INDEX)" "$f" | grep -v "IF EXISTS" && echo "DB: DROP without IF EXISTS in $f"
|
|
276
309
|
grep -nE "SELECT.*FROM" "$f" | grep -v "LIMIT" | grep -v "WHERE.*created_at" && echo "DB: possibly unbounded query in $f"
|
|
277
|
-
done
|
|
310
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep '\.sql$')
|
|
278
311
|
|
|
279
312
|
# Migration prefix collision scan (Deploy Gate — HIGH)
|
|
280
313
|
# Runs whenever any SQL file changed. Scans the full infra/migrations/ tree for duplicate
|
|
@@ -314,7 +347,8 @@ For each `.tsx`/`.ts` client component:
|
|
|
314
347
|
7. **Rules of Hooks**: Do function components avoid early returns before hook calls?
|
|
315
348
|
|
|
316
349
|
```bash
|
|
317
|
-
|
|
350
|
+
while IFS= read -r f; do
|
|
351
|
+
[ -z "$f" ] && continue
|
|
318
352
|
# Check for useEffect without cleanup (basic heuristic)
|
|
319
353
|
grep -c "useEffect" "$f" 2>/dev/null
|
|
320
354
|
grep -c "return.*cleanup\|return.*\(\) =>" "$f" 2>/dev/null
|
|
@@ -354,15 +388,16 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.tsx?$' | grep -v 'rou
|
|
|
354
388
|
}
|
|
355
389
|
in_component && /\buse[A-Z]/ { found_hook=1 }
|
|
356
390
|
' FILENAME="$f" "$f" 2>/dev/null
|
|
357
|
-
done
|
|
391
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.tsx?$' | grep -v 'route\.ts')
|
|
358
392
|
```
|
|
359
393
|
|
|
360
394
|
### 2F: Frontend proxy wiring (client-side fetch calls)
|
|
361
395
|
|
|
362
396
|
```bash
|
|
363
|
-
|
|
397
|
+
while IFS= read -r f; do
|
|
398
|
+
[ -z "$f" ] && continue
|
|
364
399
|
grep -nE '(fetch|useSWR|apiFetch)\s*[(<]\s*[`"'"'"']/api/v1/' "$f" 2>/dev/null && echo "PROXY: direct /api/v1/ call in $f — must use /api/ proxy"
|
|
365
|
-
done
|
|
400
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.(tsx?|jsx?)$' | grep -v 'route\.ts$')
|
|
366
401
|
```
|
|
367
402
|
|
|
368
403
|
### 2G: Cross-service integration (shell scripts with curl/wget)
|
|
@@ -397,7 +432,8 @@ The script `verify-host-headers.sh` checks for the `FORGE_INTERNAL_PATTERNS` env
|
|
|
397
432
|
**Why this matters**: Every DB connection function in a deploy script must use `${POSTGRES_HOST:-localhost}` (or equivalent env var interpolation), not a bare `"localhost"` string literal. A hardcoded `"localhost"` works in local development but fails in any environment where PostgreSQL runs on a separate host. The pattern is reliably grep-detectable. <!-- Added: forge#303 -->
|
|
398
433
|
|
|
399
434
|
```bash
|
|
400
|
-
|
|
435
|
+
while IFS= read -r f; do
|
|
436
|
+
[ -z "$f" ] && continue
|
|
401
437
|
# Only check files that contain DB connection tools
|
|
402
438
|
if grep -qE "PGPASSWORD|createdb|psql|pg_dump|pg_restore" "$f" 2>/dev/null; then
|
|
403
439
|
# Look for new/modified lines with hardcoded localhost in variable assignments
|
|
@@ -412,7 +448,7 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.sh$'); do
|
|
|
412
448
|
echo "$HARDCODED"
|
|
413
449
|
fi
|
|
414
450
|
fi
|
|
415
|
-
done
|
|
451
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.sh$')
|
|
416
452
|
```
|
|
417
453
|
|
|
418
454
|
Report any hit as **HIGH** — the DB connection will fail in any environment where PostgreSQL is not on `127.0.0.1:5432`.
|
|
@@ -478,7 +514,8 @@ Report each conflict as **HIGH** — a shadowed native command is a silent user-
|
|
|
478
514
|
**Severity**: MEDIUM (flags for review, not auto-blocking). The review agent's concurrency auditor verifies with full context.
|
|
479
515
|
|
|
480
516
|
```bash
|
|
481
|
-
|
|
517
|
+
while IFS= read -r f; do
|
|
518
|
+
[ -z "$f" ] && continue
|
|
482
519
|
# asyncio.shield usage — outer await may be reached with pending CancelledError
|
|
483
520
|
grep -nE "asyncio\.shield" "$f" 2>/dev/null && \
|
|
484
521
|
echo "ASYNC: asyncio.shield in $f — verify outer await isn't reachable with pending CancelledError. Prefer loop.create_task() for fire-and-forget."
|
|
@@ -493,7 +530,7 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$'); do
|
|
|
493
530
|
sed -n "$((LINENO+1)),$((LINENO+5))p" "$f" | grep -q "await" && \
|
|
494
531
|
echo "ASYNC: await in except/finally block at $f:$LINENO — may fail if CancelledError is pending"
|
|
495
532
|
done
|
|
496
|
-
done
|
|
533
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$')
|
|
497
534
|
```
|
|
498
535
|
|
|
499
536
|
|
|
@@ -516,7 +553,8 @@ For each new module-level state variable in the diff:
|
|
|
516
553
|
```
|
|
517
554
|
|
|
518
555
|
```bash
|
|
519
|
-
|
|
556
|
+
while IFS= read -r f; do
|
|
557
|
+
[ -z "$f" ] && continue
|
|
520
558
|
# Find new module-level state variables introduced in the diff
|
|
521
559
|
NEW_VARS=$(git diff HEAD -- "$f" 2>/dev/null | grep -E '^\+[A-Za-z_]\w* = (\{\}|\[\]|set\(\)|Lock\(\)|defaultdict|Counter)' | grep -oE '^[+][A-Za-z_]\w*' | tr -d '+' | sort -u)
|
|
522
560
|
for new_var in $NEW_VARS; do
|
|
@@ -533,7 +571,7 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$'); do
|
|
|
533
571
|
done
|
|
534
572
|
done
|
|
535
573
|
done
|
|
536
|
-
done
|
|
574
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$')
|
|
537
575
|
```
|
|
538
576
|
|
|
539
577
|
**Severity**: MEDIUM — missing a mutation site is usually a latent bug, not an immediate crash, but it causes incorrect behavior under concurrent load.
|
|
@@ -551,11 +589,12 @@ For each changed file in the anti-detection/scraping domains:
|
|
|
551
589
|
**1. Identify modified string identifier sets** (frozensets, tuples, or lists of string literals used for detection):
|
|
552
590
|
|
|
553
591
|
```bash
|
|
554
|
-
|
|
592
|
+
while IFS= read -r f; do
|
|
593
|
+
[ -z "$f" ] && continue
|
|
555
594
|
echo "=== String sets in $f ==="
|
|
556
595
|
# Find named constants that are frozenset/tuple/list of strings
|
|
557
596
|
grep -nE "^[A-Z_]+ = (frozenset|tuple|set)\(\[?['\"]|^[A-Z_]+ = \(['\"]|^[A-Z_]+ = \[['\"]" "$f" 2>/dev/null
|
|
558
|
-
done
|
|
597
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'anti_detection/|consumers/|browser/|shared/detection/')
|
|
559
598
|
```
|
|
560
599
|
|
|
561
600
|
**2. For each string in a modified set, check if it appears in sibling detection files:**
|
|
@@ -563,7 +602,8 @@ done
|
|
|
563
602
|
```bash
|
|
564
603
|
SERVICE_DIR=$(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E 'anti_detection/|consumers/|browser/' | head -1 | grep -oE '^services/[^/]+/[^/]+' || echo "services/worker/worker")
|
|
565
604
|
|
|
566
|
-
|
|
605
|
+
while IFS= read -r f; do
|
|
606
|
+
[ -z "$f" ] && continue
|
|
567
607
|
# Extract string values from detection sets in this file
|
|
568
608
|
STRINGS=$(grep -oE "'[a-z_][a-z0-9_-]*'" "$f" 2>/dev/null | tr -d "'" | sort -u)
|
|
569
609
|
for s in $STRINGS; do
|
|
@@ -571,13 +611,14 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'anti_
|
|
|
571
611
|
FOUND=$(grep -rl "\"$s\"\|'$s'" $SERVICE_DIR/ 2>/dev/null | grep -v "^$f$" | grep '\.py$' | head -1)
|
|
572
612
|
[ -z "$FOUND" ] && echo "STR: '$s' in $f appears NOWHERE else in $SERVICE_DIR — possible typo or dead entry"
|
|
573
613
|
done
|
|
574
|
-
done
|
|
614
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'anti_detection/|consumers/|browser/|shared/detection/')
|
|
575
615
|
```
|
|
576
616
|
|
|
577
617
|
**3. Check for identifiers mentioned in comments/docstrings of the SAME file that are NOT in the actual set** (catches the `abck` vs `_abck` class of bug):
|
|
578
618
|
|
|
579
619
|
```bash
|
|
580
|
-
|
|
620
|
+
while IFS= read -r f; do
|
|
621
|
+
[ -z "$f" ] && continue
|
|
581
622
|
echo "=== Comment/code drift in $f ==="
|
|
582
623
|
# Extract string literals from actual sets (remove quotes)
|
|
583
624
|
SET_STRINGS=$(grep -oE "frozenset\(\[([^]]+)\]\)" "$f" 2>/dev/null | grep -oE "'[^']+'" | tr -d "'" | sort -u)
|
|
@@ -589,20 +630,25 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'anti_
|
|
|
589
630
|
for ref in $COMMENT_REFS; do
|
|
590
631
|
echo "$SET_STRINGS" | grep -q "^${ref}$" || echo "STR: '$ref' mentioned in comment in $f but NOT found in any string set — possible omission or typo"
|
|
591
632
|
done
|
|
592
|
-
done
|
|
633
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'anti_detection/|consumers/|browser/|shared/detection/')
|
|
593
634
|
```
|
|
594
635
|
|
|
595
636
|
**4. Cross-reference sibling sets in related modules** (the primary inter-file check):
|
|
596
637
|
|
|
597
638
|
```bash
|
|
598
639
|
# Find all detection-related Python files in the same service
|
|
599
|
-
|
|
640
|
+
while IFS= read -r f; do
|
|
641
|
+
[ -z "$f" ] && continue
|
|
600
642
|
SIBLINGS=$(find $(dirname "$f")/../ -name "*.py" -path "*/anti_detection/*" -o -name "*.py" -path "*/consumers/*" -o -name "*.py" -path "*/browser/*" 2>/dev/null | grep -v "^$f$" | head -10)
|
|
601
643
|
|
|
602
644
|
# Extract detection set names from the changed file
|
|
603
645
|
SET_NAMES=$(grep -oE "^[A-Z_]+ = (frozenset|tuple|set)" "$f" 2>/dev/null | grep -oE "^[A-Z_]+")
|
|
604
646
|
|
|
605
|
-
|
|
647
|
+
# $SIBLINGS is one file path per line (`find`) — herestring (not a piped
|
|
648
|
+
# `| while read`, which would run the loop body in a subshell and discard
|
|
649
|
+
# any accumulator set inside it) preserves newline-safety for paths with spaces.
|
|
650
|
+
while IFS= read -r sib; do
|
|
651
|
+
[ -z "$sib" ] && continue
|
|
606
652
|
# Extract strings from sibling detection sets
|
|
607
653
|
SIB_STRINGS=$(grep -oE "'[a-z_][a-z0-9_-]*'" "$sib" 2>/dev/null | tr -d "'" | sort -u)
|
|
608
654
|
|
|
@@ -613,8 +659,8 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'anti_
|
|
|
613
659
|
echo "$SIB_STRINGS" | grep -q "^${s}$" || \
|
|
614
660
|
echo "STR: '$s' in $f not found in sibling file $sib — verify if cross-module consistency is required"
|
|
615
661
|
done
|
|
616
|
-
done
|
|
617
|
-
done
|
|
662
|
+
done <<< "$SIBLINGS"
|
|
663
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'anti_detection/|consumers/|browser/')
|
|
618
664
|
```
|
|
619
665
|
|
|
620
666
|
**Flag as findings**:
|
|
@@ -633,7 +679,8 @@ done
|
|
|
633
679
|
For each new or modified capacity constant in the diff:
|
|
634
680
|
|
|
635
681
|
```bash
|
|
636
|
-
|
|
682
|
+
while IFS= read -r f; do
|
|
683
|
+
[ -z "$f" ] && continue
|
|
637
684
|
# Find new capacity constant assignments introduced by this diff
|
|
638
685
|
NEW_CAPS=$(git diff HEAD -- "$f" 2>/dev/null | grep -E '^\+[A-Za-z_]\w*(MB|SIZE|MAX|LIMIT|THRESHOLD|TIMEOUT)\w*\s*=' | grep -oE '^[+][A-Za-z_]\w+' | tr -d '+' | sort -u)
|
|
639
686
|
for cap_var in $NEW_CAPS; do
|
|
@@ -651,7 +698,7 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'servi
|
|
|
651
698
|
git log --all -15 --oneline --grep="memory" 2>/dev/null | head -5
|
|
652
699
|
git log --all -15 --oneline --grep="$(echo $keywords | cut -d' ' -f1)" 2>/dev/null | head -5
|
|
653
700
|
done
|
|
654
|
-
done
|
|
701
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'services/worker/|infra/|browser/')
|
|
655
702
|
```
|
|
656
703
|
|
|
657
704
|
**Flag as findings**:
|
|
@@ -669,7 +716,8 @@ done
|
|
|
669
716
|
**Why this matters**: `appleboy/ssh-action` processes every `script:` field through Go's `text/template` engine **client-side, before SSH transmission**. Any `{{` in the script — including in comments, `docker ps --format` strings, and `docker inspect --format` strings — is interpreted as a Go template directive. If the expression does not resolve against the action's data context (which is an empty map for `appleboy/ssh-action`), the step exits with status 1 **before the script reaches the remote shell**. Shell error handlers (`|| fallback`, `2>/dev/null`, `set -e`) are completely bypassed. `continue-on-error: true` masks the failure silently, producing apparent deploy success with no remote shell output.
|
|
670
717
|
|
|
671
718
|
```bash
|
|
672
|
-
|
|
719
|
+
while IFS= read -r f; do
|
|
720
|
+
[ -z "$f" ] && continue
|
|
673
721
|
if grep -q "appleboy/ssh-action" "$f"; then
|
|
674
722
|
TEMPLATE_HITS=$(grep -n "{{" "$f")
|
|
675
723
|
if [ -n "$TEMPLATE_HITS" ]; then
|
|
@@ -686,7 +734,7 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.github/workflows/.*\.
|
|
|
686
734
|
echo " docker ps --format json | jq -r '.Status'"
|
|
687
735
|
fi
|
|
688
736
|
fi
|
|
689
|
-
done
|
|
737
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.github/workflows/.*\.yml$')
|
|
690
738
|
```
|
|
691
739
|
|
|
692
740
|
Report any hit as **CONFIRMED HIGH** — the step will exit 1 before reaching the remote shell.
|
|
@@ -702,7 +750,8 @@ Go template directives in `appleboy/ssh-action` `script:` blocks are preprocesse
|
|
|
702
750
|
```bash
|
|
703
751
|
PR_BASE="${PR_BASE:-staging}"
|
|
704
752
|
|
|
705
|
-
|
|
753
|
+
while IFS= read -r f; do
|
|
754
|
+
[ -z "$f" ] && continue
|
|
706
755
|
# Extract new app.* import statements added in the diff
|
|
707
756
|
# Only consider lines added (starting with +), skip diff headers (+++)
|
|
708
757
|
NEW_IMPORTS=$(git diff HEAD -- "$f" 2>/dev/null | grep -E '^\+\s*(from app\.|import app\.)' | grep -v '^+++' | sed 's/^+//')
|
|
@@ -736,7 +785,7 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$'); do
|
|
|
736
785
|
fi
|
|
737
786
|
fi
|
|
738
787
|
done
|
|
739
|
-
done
|
|
788
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$')
|
|
740
789
|
```
|
|
741
790
|
|
|
742
791
|
**Flag as findings**:
|
|
@@ -759,20 +808,24 @@ A builder agent working in a milestone-branch worktree has access to milestone-o
|
|
|
759
808
|
# Step 1: Identify the service directory from the changed Dockerfile/entrypoint path
|
|
760
809
|
# e.g. services/worker/Dockerfile → service root = services/worker/
|
|
761
810
|
SERVICE_DIR=""
|
|
762
|
-
|
|
811
|
+
while IFS= read -r f; do
|
|
812
|
+
[ -z "$f" ] && continue
|
|
763
813
|
SERVICE_DIR=$(dirname "$f")
|
|
764
814
|
break
|
|
765
|
-
done
|
|
815
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -iE 'Dockerfile|entrypoint.*\.sh')
|
|
766
816
|
|
|
767
817
|
if [ -n "$SERVICE_DIR" ]; then
|
|
768
818
|
# Step 2: Find volume mount points for this service in any docker-compose*.yml
|
|
769
819
|
COMPOSE_FILES=$(find . -maxdepth 3 -name "docker-compose*.yml" 2>/dev/null | sort)
|
|
770
820
|
VOLUME_MOUNTS=""
|
|
771
|
-
|
|
821
|
+
# $COMPOSE_FILES is one path per line (`find`) — herestring, not a piped
|
|
822
|
+
# `| while read`, so VOLUME_MOUNTS remains visible after the loop exits.
|
|
823
|
+
while IFS= read -r cf; do
|
|
824
|
+
[ -z "$cf" ] && continue
|
|
772
825
|
# Extract named volume mount points (lines like: - volume_name:/container/path)
|
|
773
826
|
MOUNTS=$(grep -oE '[a-z_-]+:/[^: ]+' "$cf" 2>/dev/null | grep -v '^#' | cut -d: -f2)
|
|
774
827
|
VOLUME_MOUNTS="$VOLUME_MOUNTS $MOUNTS"
|
|
775
|
-
done
|
|
828
|
+
done <<< "$COMPOSE_FILES"
|
|
776
829
|
|
|
777
830
|
# Step 3: Grep the service directory for filesystem write operations
|
|
778
831
|
WRITE_OPS=$(grep -rn "mkdir\|write_bytes\|open(.*['\"]w\|makedirs\|shutil\.copy\|shutil\.move" \
|
|
@@ -802,7 +855,8 @@ fi
|
|
|
802
855
|
```bash
|
|
803
856
|
PR_BASE="${PR_BASE:-staging}"
|
|
804
857
|
|
|
805
|
-
|
|
858
|
+
while IFS= read -r f; do
|
|
859
|
+
[ -z "$f" ] && continue
|
|
806
860
|
# Extract removed condition lines from the diff (lines starting with -)
|
|
807
861
|
REMOVED_CONDITIONS=$(git diff HEAD -- "$f" 2>/dev/null | grep -E '^-.*\bif\b' | grep -v '^---' | sed 's/^-//' | sed 's/^\s*//' | head -5)
|
|
808
862
|
|
|
@@ -825,7 +879,7 @@ for f in $(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'route
|
|
|
825
879
|
fi
|
|
826
880
|
done
|
|
827
881
|
fi
|
|
828
|
-
done
|
|
882
|
+
done < <(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E '\.py$' | grep -E 'router')
|
|
829
883
|
```
|
|
830
884
|
|
|
831
885
|
**Flag as findings**:
|
|
@@ -836,6 +890,72 @@ done
|
|
|
836
890
|
- Only triggers when a condition line was actually removed in the diff (not just modified)
|
|
837
891
|
- Reports file:line matches from sibling files only (excludes the fixed file itself)
|
|
838
892
|
|
|
893
|
+
### 2P: External Tool Config Schema Advisory (external tool config files)
|
|
894
|
+
|
|
895
|
+
**Triggered when**: CONFIG_SCHEMA domain is set (config files for external tools are in the diff).
|
|
896
|
+
|
|
897
|
+
**Why this matters**: External tools with strict config schemas (Traefik, nginx, Kubernetes, Terraform, Docker Compose) silently ignore or reject structurally incorrect config — no startup crash, no error log, the feature simply does not activate. The quality gate cannot run `traefik validate` / `nginx -t` / `terraform validate` (those belong in CI), but it can verify that CI validation steps exist for the tool and remind the builder that structural validation is CI's responsibility. <!-- Added: forge#1104 -->
|
|
898
|
+
|
|
899
|
+
```bash
|
|
900
|
+
# Identify which external tools have config files in the diff
|
|
901
|
+
TOOL_FILES=$(echo {CHANGED_FILES} | tr ' ' '\n' | grep -E \
|
|
902
|
+
"(traefik/|infra/nginx/|k8s/|terraform/|docker-compose.*\.yml$|.*\.(conf|toml)$)" | head -20)
|
|
903
|
+
|
|
904
|
+
if [ -n "$TOOL_FILES" ]; then
|
|
905
|
+
echo "CONFIG_SCHEMA: External tool config files detected in diff:"
|
|
906
|
+
echo "$TOOL_FILES"
|
|
907
|
+
echo ""
|
|
908
|
+
|
|
909
|
+
# Check whether CI has a validation step for each detected tool
|
|
910
|
+
WORKFLOW_DIR="{WORKTREE_PATH}/.github/workflows"
|
|
911
|
+
CI_MISSING=""
|
|
912
|
+
|
|
913
|
+
# $TOOL_FILES is one path per line (already newline-split above via `tr`) —
|
|
914
|
+
# herestring, not a piped `| while read`, so CI_MISSING remains visible
|
|
915
|
+
# after the loop exits.
|
|
916
|
+
while IFS= read -r f; do
|
|
917
|
+
[ -z "$f" ] && continue
|
|
918
|
+
case "$f" in
|
|
919
|
+
traefik/*)
|
|
920
|
+
grep -rlE "traefik.*validate|traefik.*check" "$WORKFLOW_DIR" 2>/dev/null | grep -q . || \
|
|
921
|
+
CI_MISSING="$CI_MISSING traefik"
|
|
922
|
+
;;
|
|
923
|
+
infra/nginx/*|*nginx*.conf)
|
|
924
|
+
grep -rlE "nginx.*-t|nginx.*test|nginx.*configtest" "$WORKFLOW_DIR" 2>/dev/null | grep -q . || \
|
|
925
|
+
CI_MISSING="$CI_MISSING nginx"
|
|
926
|
+
;;
|
|
927
|
+
k8s/*|*.yaml)
|
|
928
|
+
if ! grep -rlE "kubectl.*--dry-run|kubeval|kustomize.*build|helm.*lint" "$WORKFLOW_DIR" 2>/dev/null | grep -q .; then
|
|
929
|
+
echo "$f" | grep -qE '^k8s/' && CI_MISSING="$CI_MISSING kubernetes"
|
|
930
|
+
fi
|
|
931
|
+
;;
|
|
932
|
+
terraform/*)
|
|
933
|
+
grep -rlE "terraform.*validate|terraform.*plan|tflint" "$WORKFLOW_DIR" 2>/dev/null | grep -q . || \
|
|
934
|
+
CI_MISSING="$CI_MISSING terraform"
|
|
935
|
+
;;
|
|
936
|
+
esac
|
|
937
|
+
done <<< "$TOOL_FILES"
|
|
938
|
+
|
|
939
|
+
# Deduplicate
|
|
940
|
+
CI_MISSING=$(echo "$CI_MISSING" | tr ' ' '\n' | sort -u | grep -v '^$' | tr '\n' ' ')
|
|
941
|
+
|
|
942
|
+
if [ -n "$CI_MISSING" ]; then
|
|
943
|
+
for tool in $CI_MISSING; do
|
|
944
|
+
echo "CFG-1 | MEDIUM | .github/workflows/ | No CI validation step found for $tool config changes — add structural validation (e.g., 'traefik validate', 'nginx -t', 'terraform validate') to CI so structural errors are caught before merge. The INFRA review agent verifies logical correctness; CI must verify structural correctness."
|
|
945
|
+
done
|
|
946
|
+
else
|
|
947
|
+
echo "CFG: CI validation steps detected for modified tool configs — structural validation covered by CI."
|
|
948
|
+
fi
|
|
949
|
+
fi
|
|
950
|
+
```
|
|
951
|
+
|
|
952
|
+
**Flag as findings**:
|
|
953
|
+
- No CI validation step for a tool whose config files changed: **MEDIUM** — structural config errors will not be caught before deploy. The fix is a CI-layer addition (see `/ci-audit` for a comprehensive CI gap audit), not a quality gate fix.
|
|
954
|
+
|
|
955
|
+
**This check is advisory (MEDIUM), not blocking (HIGH)**. The quality gate cannot substitute for CI-level validators (`traefik validate`, `nginx -t`, `terraform validate`). Its role is to detect when those validators are absent from CI and surface that gap to the builder. Actual structural validation belongs in GitHub Actions.
|
|
956
|
+
|
|
957
|
+
**Separation of concerns**: ForgeDock reasons about architecture, logic, and security; CI runs deterministic tool validators with zero token cost. The quality gate enforces that the CI layer is complete — it does not replace it.
|
|
958
|
+
|
|
839
959
|
## Step 3: Compile findings
|
|
840
960
|
|
|
841
961
|
Format findings as a structured list:
|