@bugzy-ai/bugzy 1.17.0 → 1.18.1

package/dist/index.js

@@ -893,7 +893,29 @@ Read \`.bugzy/runtime/memory/event-history.md\` to:
 - List \`./test-cases/\` for existing tests
 - Check \`.bugzy/runtime/knowledge-base.md\` for past insights
 
-#### 2.4 Event-Action Reference Patterns
+#### 2.4 Knowledge Base: Drafted vs Real Tests (BYOT)
+
+When the project uses an external test repository, maintain two sections in \`.bugzy/runtime/knowledge-base.md\`:
+
+**Drafted Tests** (Open PRs \u2014 tests not yet merged):
+\`\`\`markdown
+### Drafted Tests
+| PR | Branch | Tests Added | Status | Date |
+|----|--------|-------------|--------|------|
+| #12 | bugzy/verify-changes-a1b2c3d4 | login.spec.ts, checkout.spec.ts | Open | 2026-02-13 |
+\`\`\`
+
+**Active Tests** (Merged \u2014 tests are part of the test suite):
+\`\`\`markdown
+### Active Tests
+| File | What it tests | Source PR | Merged |
+|------|---------------|-----------|--------|
+| login.spec.ts | Login flow with valid/invalid credentials | #12 | 2026-02-13 |
+\`\`\`
+
+Move entries from Drafted \u2192 Active Tests when PRs are merged. Remove entries when PRs are closed without merge.
+
+#### 2.5 Event-Action Reference Patterns
 
 Use these as reference patterns for common events. The webhook routing system already handles events with specific default tasks (e.g., deployment_status \u2192 /run-tests). Process-event receives events that need analysis.
 
@@ -912,6 +934,23 @@ Use these as reference patterns for common events. The webhook routing system al
 - **QA-relevant content found**: Propose appropriate follow-up tasks (e.g., \`/generate-test-cases\`, \`/verify-changes\`)
 - **No QA content** (HR meeting, offsite planning, etc.): Skip \u2014 log to event history only
 
+**External Test Repo Events** (BYOT - events from the customer's external test repository):
+- **PR opened by Bugzy** (\`com.github.external_repo.pull_request\`, \`action: "opened"\`, branch starts with \`bugzy/\`):
+  Log to Knowledge Base under "Drafted Tests". No action task needed \u2014 just record the PR number, branch, and what tests were added.
+- **PR review submitted** (\`com.github.external_repo.pull_request_review\`):
+  If changes were requested \u2192 queue \`/verify-changes\` with \`{"existingPrBranch": "{head.ref}", "context": "PR review feedback: {review.body}"}\`.
+  The execution will iterate on the existing branch, push fixes, and skip PR creation.
+- **PR comment** (\`com.github.external_repo.pull_request_comment\`):
+  Read the comment. If it contains actionable feedback, queue \`/verify-changes\`
+  with \`{"existingPrBranch": "{issue.pull_request.head.ref}"}\`.
+- **PR merged** (\`com.github.external_repo.pull_request\`, \`action: "closed"\`, \`merged: true\`):
+  Update Knowledge Base: move entries from "Drafted Tests" to "Active Tests". Notify team of new test coverage.
+  The submodule pointer update happens automatically via the container (\`updateSubmoduleToLatest\`).
+- **PR closed without merge** (\`com.github.external_repo.pull_request\`, \`action: "closed"\`, \`merged: false\`):
+  Remove from Knowledge Base "Drafted Tests". Notify team that tests were rejected.
+- **Direct push to main** (\`com.github.external_repo.push\`, ref is main/master):
+  Update Knowledge Base if test files were affected. Submodule pointer update is automatic.
+
 **Other Events:**
 - Analyze for QA relevance based on knowledge base and project context
 - If action needed, propose appropriate task. If not, log and skip.
@@ -5626,7 +5665,19 @@ After analyzing test results, triage each failure to determine if it's a product
 
 **IMPORTANT: Do NOT report bugs without triaging first.**
 
-For each failed test:
+### 1. Check Failure Classification
+
+**Before triaging any failure**, read \`new_failures\` from the latest \`test-runs/*/manifest.json\`:
+
+| \`new_failures\` State | Action |
+|------------------------|--------|
+| Non-empty array | Only triage failures listed in \`new_failures\`. Do not investigate, fix, or create issues for \`known_failures\`. |
+| Empty array | No new failures to triage. Output "0 new failures to triage" and skip the rest of this step. |
+| Field missing | Fall back: triage all failed tests (backward compatibility with older reporter versions). |
+
+### 2. Triage Each Failure
+
+For each failed test (from \`new_failures\` or all failures if field is missing):
 
 1. **Read failure details** from JSON report (error message, stack trace)
 2. **Classify the failure:**
@@ -5655,14 +5706,22 @@ For each failed test:
 - Broken navigation flows
 - Validation not working as expected
 
-**Document Classification:**
+### 3. Document Results
+
 \`\`\`markdown
-### Failure Triage
+### Failure Triage Summary
+
+**New failures triaged: N** | **Known failures skipped: M**
 
 | Test ID | Test Name | Classification | Reason |
 |---------|-----------|---------------|--------|
 | TC-001 | Login test | TEST ISSUE | Selector brittle - uses CSS instead of role |
 | TC-002 | Checkout | PRODUCT BUG | 500 error on form submit |
+
+#### Skipped Known Failures
+| Test ID | Test Name | Last Passed Run |
+|---------|-----------|-----------------|
+| TC-003 | Search | 20260210-103045 |
 \`\`\``,
   tags: ["execution", "triage", "analysis"]
 };
@@ -6220,10 +6279,36 @@ npx tsx reporters/parse-results.ts --input <file-or-url> [--timestamp <existing>
         }
       ]
     }
+  ],
+  "new_failures": [
+    {
+      "id": "<test case id>",
+      "name": "<test name>",
+      "error": "<error message or null>",
+      "lastPassedRun": "<timestamp of last passing run or null>"
+    }
+  ],
+  "known_failures": [
+    {
+      "id": "<test case id>",
+      "name": "<test name>",
+      "error": "<error message or null>",
+      "lastPassedRun": null
+    }
   ]
 }
 \`\`\`
-4. For each failed test, create:
+4. **Classify failures** \u2014 after building the manifest, classify each failed test as new or known:
+   - Read \`BUGZY_FAILURE_LOOKBACK\` env var (default: 5)
+   - List previous \`test-runs/*/manifest.json\` files sorted by timestamp descending (skip current run)
+   - For each failed test in the manifest:
+     - If it passed in any of the last N runs \u2192 \`new_failures\` (include the timestamp of the last passing run in \`lastPassedRun\`)
+     - If it failed in ALL of the last N runs \u2192 \`known_failures\`
+     - If the test doesn't exist in any previous run \u2192 \`new_failures\` (new test)
+     - If no previous runs exist at all (first run) \u2192 all failures go to \`new_failures\`
+   - Write the \`new_failures\` and \`known_failures\` arrays into the manifest
+
+5. For each failed test, create:
    - Directory: \`test-runs/{timestamp}/{testCaseId}/exec-1/\`
    - File: \`test-runs/{timestamp}/{testCaseId}/exec-1/result.json\` containing:
 \`\`\`json
@@ -6235,8 +6320,8 @@ npx tsx reporters/parse-results.ts --input <file-or-url> [--timestamp <existing>
   "testFile": "<file path if available>"
 }
 \`\`\`
-5. Print the manifest path to stdout
-6. Exit code 0 on success, non-zero on failure
+6. Print the manifest path to stdout
+7. Exit code 0 on success, non-zero on failure
 
 **Incremental mode** (\`--timestamp\` + \`--test-id\` provided):
 1. Read existing \`test-runs/{timestamp}/manifest.json\`