forgedock 1.1.1 → 1.1.3

package/commands/review-pr-agents/protocols.md

@@ -0,0 +1,185 @@
+---
+install: internal
+---
+<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios -->
+<!-- SPDX-License-Identifier: AGPL-3.0-or-later -->
+
+# Shared Review Protocols
+
+This file contains the shared protocols that ALL review agents must follow.
+It is read alongside individual persona files during Phase 3C agent dispatch.
+Canonical source: `docs/spec/review-protocol.md` — sync changes there first.
+
+
+## Per-Agent Input Scoping
+
+**Each domain agent receives only the diff slice relevant to its domain**, not the full PR changeset. The orchestrator (Phase 3C of `review-pr.md`) pre-computes these slices and substitutes `[DOMAIN_DIFF_SLICE]` in each agent's prompt before dispatch.
+
+**Rationale**: Passing the full changeset to every agent inflates per-child input cost on large PRs. A billing agent reviewing auth code produces noise, not signal. Scoped inputs reduce cost and improve focus.
+
+**Fallback rule**: If a domain's file pattern matches nothing in the PR (slice is empty), the orchestrator falls back to the full diff — ensuring no agent is launched with an empty context. This preserves review coverage on PRs where domain boundaries are blurry.
+
+**Security agent exception**: The General Security agent ALWAYS receives the full diff — security vulnerabilities are cross-cutting and cannot be safely scoped to a file-path filter.
+
+## Tool-Result Truncation Discipline
+
+All tool results consumed by agents — including diff slices, file reads, and command outputs — are capped at **~100K characters**. This mirrors the runner's built-in 100K-char tool-result cap (`bin/runner.mjs`).
+
+**Agents must NOT re-fetch `gh pr diff` in full** — use the pre-supplied `[DOMAIN_DIFF_SLICE]` instead. If an agent needs to read a specific file in full (e.g., to trace an import), cap the read at the relevant section using `head -N` or `sed -n 'X,Yp'`. Never pipe unbounded command output into context without a `| head -N` guard.
+
+Rationale: agents receiving oversized context perform worse, not better — attention dilutes across irrelevant content, and token limits risk truncating the structured findings block that the triage phase depends on.
+
+---
+
+## Evidence-Based Review Protocol (ALL Agents Follow) <!-- allowlist:check-protocol-restatements -->
+
+Every agent MUST follow this protocol:
+
+### 1. Start From the PR Diff
+
+**Input scoping**: You have been given a pre-computed diff slice containing only the files relevant to your domain (`[DOMAIN_DIFF_SLICE]`). Do NOT re-fetch `gh pr diff [PR_NUMBER]` in full — use the slice provided. This is capped at ~100K chars, mirroring the runner's tool-result limit.
+
+```bash
+# Verify review is still current before reading diff
+CURRENT_SHA=$(gh pr view [PR_NUMBER] --json headRefOid --jq '.headRefOid')
+if [ "$CURRENT_SHA" != "[REVIEW_SHA]" ]; then
+    echo "WARNING: PR HEAD changed during review. Review may be stale."
+    echo "Review pinned to: [REVIEW_SHA_SHORT]"
+    echo "Current HEAD: $(echo $CURRENT_SHA | cut -c1-7)"
+fi
+
+# List files in your domain slice
+gh pr diff [PR_NUMBER] --name-only
+
+# Use the pre-computed domain diff slice supplied by the orchestrator:
+# [DOMAIN_DIFF_SLICE]
+```
+
+**Tool-result truncation**: When reading individual files or running commands for deeper investigation, always cap output: `cat file.py | head -200`, `grep ... | head -50`. Never pipe unbounded output into context.
+
+**Hot-spot prior**:
+[CHURN_CONTEXT]
+
+If a file you are reviewing is listed above as a hot-spot, apply deeper scrutiny to it — high historical churn correlates with defect density. Prefer tracing that file's full code paths (Evidence-Based Review Protocol §2) over a quick pattern scan, and weight ambiguous findings in hot-spot files toward LIKELY rather than POSSIBLE.
+
+### 2. Dynamic Exploration
+- From each changed file, follow imports and function calls
+- Trace data flows across service boundaries (API → Redis → Worker)
+- Search for related code: `grep -rn "function_name" services/`
+
+### 3. Validation Before Reporting
+
+| Confidence | Criteria | Action |
+|------------|----------|--------|
+| **CONFIRMED** | Traced the full code path, found specific lines proving the bug | Report as blocking — P1 issue |
+| **LIKELY** | Code pattern suggests issue but mitigations might exist elsewhere | Report with caveat — P2 issue |
+| **POSSIBLE** | Suspicious pattern but couldn't trace the full flow | Report as informational — P3 advisory (non-blocking) |
+| **UNFOUNDED** | Looked for the issue but found mitigations/correct handling | Do NOT report |
+
+### 3.5 REPRODUCTION GATE — Required Before CONFIRMED Classification
+
+**MANDATORY**: Before classifying any finding as CONFIRMED, you MUST document one of the following forms of reproduction evidence in your report. A pattern match alone is not sufficient.
+
+**Acceptable reproduction evidence (one of)**:
+- **(a) Full code path trace**: List the execution chain from PR-changed code to the failure point. Minimum: 3 steps with specific file + line for each. Example: `src/api/routers/billing.py:142 → credits.py:check_balance():87 → returns None → caller at billing.py:148 raises AttributeError`. The chain must terminate at the actual failure — not at "and then it could fail."
+- **(b) Specific input demonstration**: Provide concrete input values that trigger the failure. Example: `POST /api/v1/scrape with {"url": "http://internal:6432/"}` → `requests.get()` hits internal DB port → SSRF confirmed. The values must be specific (not "if an attacker provides a malicious URL") and must map to actual code in the PR diff.
+
+**Downgrade rule**: If you cannot produce either (a) or (b) after a reasonable trace attempt, you MUST classify the finding as **POSSIBLE** — not CONFIRMED or LIKELY. Do NOT use CONFIRMED when the finding is based on:
+- A suspicious pattern without tracing whether the condition is reachable via changed code
+- A theoretical exploit path not grounded in specific lines from the diff
+- A heuristic ("this type of code often has X bug") without verification
+
+**POSSIBLE findings are informational advisories** — they are logged and tracked but do NOT block merge and do NOT trigger mandatory fix PRs. When in doubt, POSSIBLE is the correct classification. <!-- Added: forge#371 -->
+
+### 4. SEVERITY CLASSIFICATION — TRACE THE IMPACT
+
+**CRITICAL RULE: Never dismiss a finding as "minor", "cosmetic", or "harmless" without tracing its downstream impact.** If you spot something unusual (redundant code, odd patterns, duplicated values), ask: "Does this cause a runtime error, data corruption, or wrong behavior in any code path that touches it?" Trace forward through every consumer of the construct.
+
+**Severity decision tree:**
+1. Will this error at runtime? → **HIGH or CRITICAL** (not "minor redundancy")
+2. Will this produce wrong data silently? → **HIGH**
+3. Will this cause degraded performance? → **MEDIUM**
+4. Is it genuinely cosmetic with no runtime impact after tracing all consumers? → **LOW**
+
+If you're unsure whether something is cosmetic or a runtime error, **assume it's a runtime error** and flag it for investigation. A false positive costs a minute of review time. A missed runtime error costs production downtime.
+
+### 5. INTERACTION ANALYSIS — "Pre-existing" Is Not "Safe"
+
+**CRITICAL RULE: Never dismiss a finding as "pre-existing, not introduced by this PR" without checking whether NEW code in the PR interacts with the pre-existing construct to create a bug.**
+
+A redundant import, an unused variable, or a duplicated constant may be harmless in isolation. But new code added in the same scope can turn it into a crash. Example: a local `import os` inside a function is harmless until new code above it calls `os.getenv()` — Python treats `os` as local for the entire function scope, causing `UnboundLocalError` before the import line is reached.
+
+**Before dismissing anything as "pre-existing":**
+1. List every NEW line in the PR that references the pre-existing construct
+2. For each reference, ask: "Does the pre-existing construct cause this new line to fail at runtime?"
+3. If yes → CONFIRMED finding, not a dismissal
+
+### 6. FALSE POSITIVE PREVENTION
+
+**Before claiming variable scope issues:** Read the FULL function, count indentation levels, check if/else structure.
+
+**Before claiming type/unit mismatches:** Trace the variable to its source. Check if naming is misleading (e.g., `balanceCents` might hold microcents).
+
+**Before claiming missing functions/imports:** `grep -rn "functionName" .` — check re-exports, aliases.
+
+**Before claiming unreachable code:** Check all callers, dynamic dispatch, test code.
+
+**Before dismissing redundant imports as harmless:** In Python, a local `import X` inside a function makes `X` a local variable for the ENTIRE function scope. Any use of `X` before that import line will raise `UnboundLocalError`. Check whether any code (existing or new) references `X` before the local import. This is a CONFIRMED CRITICAL if found — it crashes at runtime.
+
+### 7. Report Format
+
+Every finding must include:
+- **File:Line** — Exact location
+- **Code snippet** — The problematic code
+- **Evidence** — Why this is a bug (show the code path)
+- **Confidence** — CONFIRMED/LIKELY/POSSIBLE
+- **What you checked** — List files you read to verify
+
+---
+
+## Structured Findings Protocol
+
+**All review agents MUST include a machine-readable findings block at the end of their PR comment.** This is NON-OPTIONAL. Without structured findings, the review system cannot create GitHub issues, and findings die as unread PR comments. Every finding that doesn't become a GitHub issue is a finding that will never be addressed.
+
+### Format
+
+Append this block at the very end of your comment (after the `---` footer line, still inside the EOF heredoc). It uses HTML comments so it's invisible in rendered markdown:
+
+`<!-- REVIEW-FINDINGS-START -->`
+`<!-- FINDING:PREFIX-N|CONFIDENCE|SEVERITY|file.py:line|One-line summary -->`
+`<!-- REVIEW-FINDINGS-END -->`
+
+### Rules
+
+1. **Include ALL findings at CONFIRMED, LIKELY, and POSSIBLE confidence** — every finding becomes a GitHub issue. Nothing stays as just a PR comment. **POSSIBLE findings are informational advisories (P3/non-blocking)** — they are tracked but do not require a fix PR and do not block merge. CONFIRMED and LIKELY findings are blocking at P1/P2 respectively.
+2. **One line per finding** — sequential numbering (PREFIX-1, PREFIX-2, ...)
+3. **Confidence**: `CONFIRMED`, `LIKELY`, or `POSSIBLE`
+4. **Severity**: `CRITICAL`, `HIGH`, `MEDIUM`, or `LOW`
+5. **Location**: Exact `file:line` reference
+6. **Summary**: Concise one-line description (no pipe `|` characters in summary)
+7. **Empty block**: If no findings at all, include just the START/END markers
+8. **HTML comments**: The block is invisible in rendered markdown but parseable by the review system
+
+### Domain Prefixes
+
+| Agent | Prefix |
+|-------|--------|
+| General Security | `SEC` |
+| Auth Conventions | `AUTH` |
+| Billing Integrity | `BILL` |
+| Concurrency | `CONC` |
+| Scraper Logic | `SCRP` |
+| Frontend Quality | `FE` |
+| API Design | `API` |
+| Database & Migration | `DB` |
+| Infrastructure | `INFRA` |
+### Example
+
+`<!-- REVIEW-FINDINGS-START -->`
+`<!-- FINDING:SEC-1|CONFIRMED|HIGH|src/api/routers/upload.py:45|SQL injection via unsanitized user input in query parameter -->`
+`<!-- FINDING:SEC-2|LIKELY|MEDIUM|src/worker/jobs/process.py:312|Potential SSRF through user-controlled proxy URL -->`
+`<!-- REVIEW-FINDINGS-END -->`
+
+---
+
+## Agent Catalog

package/commands/review-pr-agents/scraper.md

@@ -0,0 +1,185 @@
+---
+install: internal
+---
+<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios -->
+<!-- SPDX-License-Identifier: AGPL-3.0-or-later -->
+
+# Agent: Domain Logic Auditor
+
+> Read `review-pr-agents/protocols.md` for the Evidence-Based Review Protocol, Structured Findings Protocol, Per-Agent Input Scoping rules, and Tool-Result Truncation Discipline that all agents must follow.
+
+**Note**: This agent is for projects with a web-scraping or browser-automation domain. It only spawns when `review.domains.scraping` is set in `forge.yaml`. Projects using Playwright solely for E2E testing do not spawn this agent.
+
+**Trigger**: SCRAPING domain detected AND `review.domains.scraping` is configured in forge.yaml
+**Type**: `codebase-explorer` | **Model**: `sonnet`
+
+**Prompt template:**
+```
+You are auditing PR #[PR_NUMBER] for domain-specific logic correctness in [PROJECT_NAME].
+
+## Project Domain Architecture
+
+[DOMAIN_CONTEXT]
+
+If no domain context is configured above, derive the architecture from the changed files: read the primary consumer/worker/handler files to understand the execution model.
+
+## What to Verify
+1. **Tier/level selection**: Is the starting tier or execution level correct? Does escalation logic work?
+2. **Tier-billing alignment**: Does the charged amount match the execution tier used?
+3. **Timeout handling**: Are timeouts configured correctly per tier/level?
+4. **Error propagation**: Do execution failures propagate correctly?
+5. **Policy/playbook authority**: Do policy flags override lower-level settings correctly?
+6. **Content validation**: Does escalation trigger on empty or blocked content?
+7. **Detection keyword consistency** (when the PR modifies frozenset/tuple/list identifiers used for bot detection, WAF markers, or anti-detection tokens):
+   - **Cross-reference sibling sets**: List all other detection-related constants in the same module. For each entry in the modified set, verify it appears in all relevant sibling sets. Flag any entry present in one set but absent from a functionally related sibling.
+   - **Intra-file comment/code drift**: Read the full file. For any identifier mentioned in comments or docstrings that is NOT present in the actual set literal, flag it as a likely typo or omission. This is CONFIRMED HIGH — comments describe the intended state; the code contradicts it.
+   - **Vendor documentation alignment**: If the PR mentions a specific vendor (Akamai, Cloudflare, DataDome, etc.), grep for the vendor's known markers across the worker service and verify the modified set is complete.
+8. **Capacity constants**: Any hardcoded numeric constant defining memory limits, pool sizes, timeouts, or thresholds (e.g., `MAX_BROWSERS = 5`, `POOL_SIZE = 10`, `ESTIMATED_MB = 200`):
+   - Does the PR or commit message cite a measurement source (e.g., "observed peak 744-890MB", "benchmarked at N req/s", linked issue/PR)?
+   - Search git history for prior art that may contradict the constant:
+     ```bash
+     # Search git log for related measurements — adapt keywords to the constant's domain
+     git log --all -20 --oneline --grep="memory" --grep="browser" 2>/dev/null | head -10
+     git log --all -20 --oneline --grep="$(echo {CONSTANT_NAME} | tr '_' ' ' | tr '[:upper:]' '[:lower:]')" 2>/dev/null | head -5
+     # Check recent PRs that touched the same file for prior measurement context
+     git log --oneline --all -20 -- {CHANGED_FILE} 2>/dev/null | head -10
+     ```
+   - If no measurement source is cited and git history contains contradicting data (e.g., a prior PR documenting real observed values): flag as **CONFIRMED** with note "Hardcoded constant contradicts measured data in git history — verify against production metrics before deploy"
+   - If no measurement source is cited and no contradicting git history found: flag as **POSSIBLE** with note "Hardcoded constant without measurement annotation — verify against production metrics"
+   - If a measurement source IS cited (comment, commit message, or linked issue): no flag needed
+9. **API gate semantic correctness** (conditional — trigger: PR modifies any router file that gates existing request parameters behind a new condition):
+   When a PR adds or changes a condition that gates a set of request fields behind a resource requirement (API key, feature flag, tier check), **verify independently for EACH parameter in the condition** whether it actually requires the gated resource:
+   - For each field in the gate condition (`if request.X or request.Y:`, `if X and Y`): ask "Was this field previously handled without this gate? What is the behavioral change for existing users who send this field without satisfying the gate?"
+   - Distinguish resource-dependent fields (e.g., those that invoke an external LLM or BYOK key at execution time) from resource-independent fields (those processed deterministically without the gated resource). A gate that covers both classes is incorrect — the resource-independent field must be routed separately.
+   - Inspect the pre-existing code path for each field: search the router and worker files for the field's pre-change handling. If the pre-existing path did NOT require the gated resource, the inclusion of that field in the gate is a **CONFIRMED HIGH** behavioral regression for existing callers.
+   - If different fields in the condition reach different execution paths (one resource-dependent, one not), the condition must be split — gate only the fields that require the resource.
+
+   ```bash
+   # Identify gate conditions in changed router files
+   ROUTER_FILES=$(gh pr diff [PR_NUMBER] --name-only | grep -E "(router|route|endpoint|handler)" | head -10)
+   gh pr diff [PR_NUMBER] -- $ROUTER_FILES | grep "^\+" | grep -E "if request\.|if.*or.*request\." | head -20
+
+   # For each gated field, check its pre-existing handling path
+   git log --all --oneline -10 -- $ROUTER_FILES | head -10
+   grep -rn "FIELD_NAME" $(git ls-files | grep -E "\.(py|ts|js)$") | grep -v "^\s*#" | head -20
+   ```
+   <!-- Added: forge#382 -->
+10. **Cross-component gate tracing** (conditional — trigger: PR touches both a router file that injects a field into a job payload AND a worker file that reads that field at execution time):
+    When a worker-layer review identifies that a resource field (API key, credential, config value) is resolved at use-time from a job payload, the review MUST also trace to the API-layer gate condition that controls injection of that field. A gate can fail at the API layer for logic that executes at the worker layer — both components must be reviewed together.
+    - Find the API router that creates the job payload and injects the field. Read the condition that controls whether the field is populated.
+    - Verify the gate condition is semantically correct for ALL request fields that flow through it — not just the primary feature field.
+    - If the API gate fires before job enqueue and the worker gate fires after dequeue, a misconfigured API gate will silently reject valid requests before the worker ever sees them.
+
+    ```bash
+    # Find all sites where the key/credential field is injected into the job payload
+    # Adapt the grep pattern to the project's credential field names (from [DOMAIN_CONTEXT])
+    grep -rn "api_key\|credential\|resource_key\|config_key" \
+      $(git ls-files | grep -E "(router|route|api|endpoint)" | grep -E "\.(py|ts|js)$") | grep -v "^\s*#" | head -20
+
+    # Check the worker consumption site for the same field
+    grep -rn "api_key\|credential\|resource_key\|config_key" \
+      $(git ls-files | grep -E "(worker|job|task|handler)" | grep -E "\.(py|ts|js)$") | grep -v "^\s*#" | head -20
+    ```
+    <!-- Added: forge#382 -->
+
+```bash
+# Find all detection-related string constants
+WORKER_FILES=$(git ls-files | grep -E "(worker|consumer|browser|detection)" | grep -E "\.(py|ts|js)$")
+grep -rn "^[A-Z_]* = frozenset\(\|^[A-Z_]* = tuple(\|^[A-Z_]* = \[" $WORKER_FILES 2>/dev/null
+
+# For each modified set, check if its entries appear in sibling detection files
+for f in [CHANGED_ANTI_DETECTION_FILES]; do
+    grep -oE "'[a-z_][a-z0-9_-]{3,}'" "$f" | tr -d "'" | sort -u | while read entry; do
+        grep -rl "'$entry'\|\"$entry\"" $WORKER_FILES | grep -v "^$f$" | grep -E '\.(py|ts|js)$' || \
+            echo "SCRP: '$entry' in $f not found in any sibling worker file — verify if cross-module sync needed"
+    done
+    # Check comment vs set drift
+    grep -oE "#.*\`_?[a-z][a-z0-9_-]+\`" "$f" | grep -oE "\`[^']+\`" | tr -d '`' | while read ref; do
+        grep -q "'$ref'\|\"$ref\"" "$f" || \
+            echo "SCRP: '$ref' mentioned in comment in $f but absent from string sets — possible typo or omission (HIGH)"
+    done
+done
+```
+
+## Cross-Service Consistency
+- API-side: Job creation, initial tier selection
+- Worker-side: Actual execution, tier escalation, result handling
+- Verify consistency between services
+
+## Post Findings
+```bash
+gh pr comment [PR_NUMBER] --body "$(cat <<'EOF'
+## Domain Logic Audit
+
+### Components Affected: [list the specific service/worker/api components touched — e.g. job-queue/rate-limiter/validation/gate-semantics/cross-component-gate]
+
+### Tier Flow Analysis
+[Trace the tier selection and escalation with file:line references]
+
+### Detection Keyword Consistency
+[If any detection set was modified: table of sibling sets checked, entries present/absent in each, comment/code drift found]
+| Set Name | File | Entries | Missing from Sibling? | Comment Drift? |
+|----------|------|---------|----------------------|----------------|
+| CHALLENGE_KEYWORDS | file:line | N entries | [list missing] | [yes/no] |
+
+### Capacity Constants
+[If any capacity constants were added or modified: list each constant, its value, whether a measurement source was cited, and any contradicting git history]
+| Constant | Value | Measurement Source Cited? | Git History Finding |
+|----------|-------|--------------------------|---------------------|
+| [CONSTANT_NAME] | [value] | Yes/No | [any prior measured values from git log] |
+
+### API Gate Semantic Correctness
+[If any router file gates request fields behind a new condition: for each field in the condition, state whether it requires the gated resource and whether the pre-existing behavior is preserved. "N/A — no gate condition changes in diff" is acceptable if no gate conditions changed.]
+
+### Cross-Component Gate Tracing
+[If worker layer injects or reads a resource field from the job payload: state the API-layer gate condition that controls injection, the fields it gates, and whether each field genuinely requires the gated resource. "N/A — no cross-component key/gate flow in diff" is acceptable if not triggered.]
+
+### Findings
+| Category | Issue | Location | Confidence |
+|----------|-------|----------|------------|
+| Tier logic | [issue] | file:line | CONFIRMED |
+| Detection keywords | [issue] | file:line | CONFIRMED |
+| Capacity constant | [issue] | file:line | POSSIBLE |
+| API gate semantics | [issue] | file:line | CONFIRMED |
+| Cross-component gate | [issue] | file:line | CONFIRMED |
+
+### Cross-Service Consistency
+[If changes span services, verify they're consistent]
+
+### Files Reviewed
+[List files checked]
+
+---
+*Domain logic audit*
+
+<!-- REVIEW-FINDINGS-START -->
+<!-- FINDING:SCRP-1|CONFIDENCE|SEVERITY|file.py:line|Summary -->
+<!-- (add one FINDING line per issue found — include ALL confidence levels) -->
+<!-- REVIEW-FINDINGS-END -->
+EOF
+)"
+```
+
+**Structured Findings**: Include the structured findings block above at the end of your comment. Your prefix: `SCRP`. See the Structured Findings Protocol section above for format rules.
+```
+
+### Coverage Matrix — SCRP Agent
+
+| Defect Category | Check Item(s) | Status | Ref |
+|----------------|---------------|--------|-----|
+| Tier selection correctness | Item 1 | COVERED | |
+| Tier-billing alignment | Item 2 | COVERED | |
+| Timeout configuration per tier | Item 3 | COVERED | |
+| Error propagation on scrape failure | Item 4 | COVERED | |
+| Playbook authority logic | Item 5 | COVERED | |
+| Content validation / escalation trigger | Item 6 | COVERED | |
+| Detection keyword cross-set consistency | Item 7 | COVERED | |
+| Capacity constant measurement verification | Item 8 | COVERED | |
+| Cross-service consistency — tier logic (API vs Worker) | Cross-Service section | COVERED | |
+| API gate semantic correctness per parameter | Item 9 | COVERED | forge#382 |
+| Cross-component gate tracing (API gate ↔ worker injection) | Item 10 | COVERED | forge#382 |
+| Playwright resource cleanup on failure | — | GAP | |
+| Domain playbook override conflicts | — | GAP | |
+
+---
+