@wazir-dev/cli 1.2.0 → 1.4.0

package/docs/research/2026-03-20-agents/a1d0ac79ac2f11e6f.txt

@@ -0,0 +1,2 @@
+{"parentUuid":null,"isSidechain":true,"promptId":"67fcde4a-7483-4526-a2da-9ae9514890c8","agentId":"a1d0ac79ac2f11e6f","type":"user","message":{"role":"user","content":"Research ONLY — no code. How do systems turn runtime findings into permanent rules?\n\nResearch:\n1. SonarQube — how do quality profiles evolve? Can findings become new rules?\n2. ESLint — how are custom rules created from recurring patterns?\n3. GitHub Advanced Security — how do vulnerability patterns become detection rules?\n4. Snyk — how do dependency vulnerability patterns get added to the database?\n\nFor Wazir: the learning loop is:\n- Review finding: \"executor missed error handling in async code\"\n- Recurs 3+ times across runs\n- Becomes antipattern: AP-XX in expertise/antipatterns/\n- Loaded into future reviewer context\n\nResearch: how should this promotion work?\n- Automatic (risky — bad patterns could become rules)?\n- Human-approved (current design)?\n- Semi-automatic (auto-propose, auto-accept after N confirmations)?\n\nOutput: Finding-to-antipattern promotion pipeline design with concrete criteria."},"uuid":"a5799425-d9aa-406f-a824-99da4ed014c2","timestamp":"2026-03-20T13:34:11.863Z","userType":"external","entrypoint":"cli","cwd":"/Users/mohamedabdallah/Work/Wazir","sessionId":"0c5a9bfa-6816-42b1-a97d-37ae7dad26c5","version":"2.1.80","gitBranch":"feat/remaining-6-fixes","slug":"mighty-wondering-giraffe"}
+{"parentUuid":"a5799425-d9aa-406f-a824-99da4ed014c2","isSidechain":true,"agentId":"a1d0ac79ac2f11e6f","message":{"model":"claude-opus-4-6","id":"msg_01F29tUsish8VbKJ6bqWuCKj","type":"message","role":"assistant","content":[{"type":"text","text":"\n\n# Research: Finding-to-Rule Promotion Pipelines\n\nI'll research how major tools handle the promotion of runtime findings into permanent rules, then design a concrete pipeline for Wazir.\n\n---\n\n## 1. SonarQube — Quality Profile Evolution\n\n**How it works:**\n\nSonarQube ships with a large built-in rule catalog per language. Organizations customize behavior through **Quality Profiles**, which are named collections of activated rules with configured severities and parameters.\n\n**Can findings become new rules?**\n\n- **Not directly.** SonarQube does not auto-generate rules from detected issues. The rule catalog is maintained by SonarSource engineers and the community.\n- **Custom rules** can be written as plugins (Java-based for most analyzers) or via XPath/regex patterns for XML-based languages.\n- The closest mechanism is **\"issue templates\"** — parameterized rules that admins can instantiate with specific patterns (e.g., \"ban calls to `Thread.sleep()` in production code\").\n- Quality Profiles evolve through: (a) SonarSource releasing new rules in upgrades, (b) admins activating/deactivating rules, (c) admins adjusting severity thresholds.\n\n**Key insight:** SonarQube treats rule creation as a **human-curated, release-gated process**. There is no automated promotion from \"frequently seen issue\" to \"new rule.\" The signal that a pattern deserves a rule comes from human analysis of recurring findings across projects.\n\n---\n\n## 2. ESLint — Custom Rules from Recurring Patterns\n\n**How it works:**\n\nESLint's architecture explicitly supports rule creation by end users:\n\n- Each rule is a single JS/TS module that exports a `create()` function receiving an AST context.\n- Rules are packaged as **plugins** (`eslint-plugin-*`) and activated in config files.\n- The ecosystem has hundreds of community plugins born from recurring patterns people noticed in code review.\n\n**The typical lifecycle of a pattern becoming a rule:**\n\n1. A team notices the same code review comment appearing repeatedly (e.g., \"don't use `forEach` with `async` callbacks\").\n2. Someone writes a custom rule that walks the AST looking for that pattern.\n3. The rule is tested against the existing codebase to calibrate false-positive rates.\n4. It's added to the team's shared config.\n5. If broadly useful, it gets published as a plugin or proposed to eslint-core.\n\n**Key insight:** ESLint's model is **human-initiated, community-validated**. The tooling makes rule authoring easy (AST visitor pattern, test harness), but a human must decide \"this pattern is worth codifying.\" The quality gate is: does it have a low false-positive rate across real codebases?\n\n---\n\n## 3. GitHub Advanced Security — Vulnerability Pattern Detection\n\n**How it works:**\n\nGitHub Advanced Security uses two systems:\n\n- **CodeQL** — a semantic code analysis engine where vulnerability patterns are expressed as **queries** (declarative logic over a relational database extracted from code).\n- **Secret scanning** — regex-based pattern matching for credentials, with patterns contributed by partner programs.\n\n**How patterns become detection rules:**\n\n- **CodeQL queries** are maintained in the `github/codeql` repository. New queries follow a strict process:\n  1. A security researcher identifies a vulnerability class.\n  2. They write a CodeQL query with precision/recall metadata.\n  3. The query goes through peer review with mandatory **false-positive benchmarking** against open-source corpora.\n  4. Queries are tagged with precision levels: `high`, `medium`, `low`. Only `high` and `medium` precision queries ship in the default suite.\n  5. Community contributions go through the same review bar.\n\n- **Secret scanning patterns** are added when a service provider registers their token format with GitHub's partner program, or when GitHub's security team identifies a new credential pattern.\n\n**Key insight:** GitHub's model is **expert-authored, precision-gated**. The critical quality gate is false-positive rate measured against real-world benchmarks. Queries with >5% false-positive rate on the benchmark corpus are not included in the default suite.\n\n---\n\n## 4. Snyk — Vulnerability Database Evolution\n\n**How it works:**\n\nSnyk maintains a proprietary vulnerability database that aggregates from multiple sources:\n\n- **Upstream feeds:** NVD (CVE), vendor advisories, language-specific advisory databases (RubyGems, npm, PyPI advisories).\n- **Snyk Security Research Team:** Manually discovers, validates, and enriches vulnerability records.\n- **Community reports:** Users can report vulnerabilities through Snyk's disclosure program.\n\n**The promotion pipeline:**\n\n1. **Signal detection:** Automated monitors watch upstream feeds, commit messages, changelogs, and issue trackers for vulnerability signals.\n2. **Triage:** Automated classifiers sort signals by likely severity and relevance.\n3. **Human validation:** Security analysts verify the vulnerability, write the advisory, determine affected version ranges, and assign severity (CVSS).\n4. **Enrichment:** Analysts add exploit maturity ratings, fix guidance, and remediation paths.\n5. **Publication:** The entry enters the live database and triggers alerts for affected users.\n\n**Key insight:** Snyk's model is **machine-detected, human-validated**. Automation handles the high-volume signal gathering, but every entry that affects user decisions goes through human review. The quality gate is analyst verification of exploitability and affected versions.\n\n---\n\n## Synthesis: The Three Models\n\n| Model | Example | Signal Source | Gate | Risk |\n|---|---|---|---|---|\n| **Fully human** | SonarQube rules | Human observation | Expert authorship + release cycle | Slow to adapt |\n| **Human-initiated, community-validated** | ESLint plugins | Recurring review comments | FP rate on real code | Inconsistent quality across plugins |\n| **Machine-detected, human-validated** | Snyk DB, GitHub CodeQL | Automated scanning + heuristics | Human analyst verification | Bottleneck at review stage |\n\nNo major system uses **fully automatic** promotion. The universal lesson: **false positives are more damaging than false negatives** in rule systems, because every false positive erodes trust and causes rule fatigue.\n\n---\n\n## Wazir Design: Finding-to-Antipattern Promotion Pipeline\n\n### The Problem\n\nWazir's review loop generates findings like:\n\n> \"executor missed error handling in async code\"\n\nWhen this recurs 3+ times, it should become a codified antipattern (`AP-XX`) loaded into future reviewer context. The question is how to manage that promotion reliably.\n\n### Recommended Model: Semi-Automatic (Auto-Propose, Human-Approve)\n\nThis is the Snyk/CodeQL hybrid model adapted for Wazir's context. Fully automatic is too risky (bad pattern becomes a permanent rule). Fully manual is too slow (findings accumulate without action). Semi-automatic captures the best of both.\n\n### Pipeline Stages\n\n```\nStage 1: TALLY          Stage 2: CANDIDATE       Stage 3: PROMOTE         Stage 4: ACTIVE\n(automatic)              (automatic)              (human gate)             (automatic)\n\nFinding logged    -->   Recurrence detected  --> Candidate proposed   --> Antipattern loaded\nin run state            and grouped               for human review         into reviewer context\n```\n\n### Stage 1 — TALLY (Automatic)\n\n**What happens:** Every review finding is logged to the run's state DB with structured metadata.\n\n**Schema per finding:**\n\n```yaml\nfinding_id: \"f-20260320-001\"\nrun_id: \"run-20260320-010128\"\nphase: \"implement\"\nrole: \"reviewer\"\ncategory: \"error-handling\"       # from a fixed taxonomy\npattern_text: \"missing error handling in async code\"\nfile_path: \"src/executor/runner.ts\"\ncode_snippet_hash: \"<sha256 of normalized snippet>\"\nseverity: \"major\"\n```\n\n**Normalization:** Before storage, the finding text is normalized — file paths, variable names, and line numbers are stripped to produce a **canonical pattern string**. This enables matching across runs.\n\n### Stage 2 — CANDIDATE (Automatic)\n\n**What happens:** After each run completes, a post-run analysis step groups findings by canonical pattern and checks recurrence thresholds.\n\n**Promotion criteria (all must be met):**\n\n| Criterion | Threshold | Rationale |\n|---|---|---|\n| Recurrence count | >= 3 distinct runs | Filters one-off mistakes |\n| Distinct files | >= 2 different files | Confirms it's a pattern, not a single-file issue |\n| Time span | >= 2 distinct calendar days | Prevents burst-counting from a single session |\n| Not already an antipattern | No existing AP with >0.8 similarity | Avoids duplicates |\n| Not already rejected | Not in the rejection log | Respects prior human decisions |\n\n**Output:** A candidate record written to `~/.wazir/projects/<slug>/candidates/`:\n\n```yaml\ncandidate_id: \"cand-20260320-001\"\ncanonical_pattern: \"missing error handling in async code\"\ncategory: \"error-handling\"\noccurrences:\n  - { run: \"run-20260318-...\", file: \"src/a.ts\", finding: \"f-...\" }\n  - { run: \"run-20260319-...\", file: \"src/b.ts\", finding: \"f-...\" }\n  - { run: \"run-20260320-...\", file: \"src/c.ts\", finding: \"f-...\" }\nproposed_severity: \"major\"          # mode of occurrence severities\nproposed_title: \"AP-XX: Unhandled errors in async code paths\"\nproposed_body: |                    # auto-drafted from occurrences\n  ## Antipattern\n  Async functions or promise chains without try/catch or .catch() handlers.\n  \n  ## Why it matters\n  Unhandled rejections crash the process or silently swallow errors.\n  \n  ## Correct pattern\n  Wrap async calls in try/catch; attach .catch() to every promise chain.\n  \n  ## Evidence\n  Found in 3 runs across 3 files over 2 days.\nstatus: \"pending_review\"\ncreated_at: \"2026-03-20T12:00:00Z\"\n```\n\n### Stage 3 — PROMOTE (Human Gate)\n\n**What happens:** The human (or a designated approver role) reviews candidates.\n\n**Review interface:** A CLI command:\n\n```\nwazir antipatterns review\n```\n\nThis presents each pending candidate with its evidence and offers three actions:\n\n| Action | Effect |\n|---|---|\n| **approve** | Candidate becomes an antipattern file in `expertise/antipatterns/` |\n| **reject** | Candidate is logged to rejection list; same pattern won't be re-proposed |\n| **defer** | Candidate stays pending; re-surfaces after N more occurrences (default: +2) |\n\n**On approve:**\n- A new file `expertise/antipatterns/AP-XX-<slug>.md` is created from the candidate body.\n- The antipattern is added to the antipattern index.\n- The candidate is marked `promoted`.\n\n**On reject:**\n- The candidate's canonical pattern is added to `~/.wazir/projects/<slug>/candidates/rejected.yaml`.\n- A reason field is recorded (optional but encouraged).\n\n### Stage 4 — ACTIVE (Automatic)\n\n**What happens:** Antipatterns in `expertise/antipatterns/` are loaded into reviewer context at the start of every review phase. No additional action needed — this already works in Wazir's current design.\n\n**Feedback loop:** If a promoted antipattern stops appearing in findings for N consecutive runs (suggested: 10), it is flagged as \"possibly resolved\" for the human to archive or keep.\n\n### Quality Safeguards\n\n1. **No fully automatic promotion.** The human gate at Stage 3 prevents bad patterns from becoming permanent rules. This is the universal lesson from SonarQube, ESLint, CodeQL, and Snyk.\n\n2. **Rejection memory.** Rejected patterns are remembered so the system doesn't nag. This avoids the anti-pattern of every tool that re-suggests the same dismissed finding.\n\n3. **Similarity matching at proposal time.** New candidates are compared against existing antipatterns (simple text similarity on canonical pattern + category match). Threshold of 0.8 prevents near-duplicate antipatterns from accumulating.\n\n4. **Decay/archival.** Antipatterns that stop triggering are surfaced for human review. Rules that outlive their usefulness get cleaned up instead of accumulating forever.\n\n5. **Evidence-linked.** Every antipattern traces back to specific findings in specific runs. This supports auditing (\"why does this rule exist?\") and makes it possible to revoke a rule if its evidence turns out to be flawed.\n\n### Comparison to Alternatives\n\n**Why not fully automatic?**\n\nThe risk is a feedback loop: a false finding recurs because of a systematic misunderstanding by the reviewer LLM (not an actual code problem), gets promoted to an antipattern, then reinforces the same misunderstanding in future reviews. This is the \"model training on its own errors\" problem. The human gate breaks this loop.\n\n**Why not fully manual?**\n\nWazir generates findings at a volume where manual tracking is impractical. The auto-tally and auto-candidate stages handle the bookkeeping. The human only needs to make a yes/no/defer decision on pre-filtered, pre-drafted candidates — a much lighter cognitive load than \"go through all findings and decide which ones deserve codification.\"\n\n**Why 3 occurrences and not 2 or 5?**\n\n- At 2: too many false promotions from coincidental pairs.\n- At 5: too slow to codify genuine patterns, especially in smaller projects.\n- 3 across 2+ files and 2+ days is the sweet spot from the CodeQL and Snyk research — it filters noise while remaining responsive.\n\n### Summary\n\nThe recommended design is a four-stage pipeline: automatic tallying, automatic candidate extraction at threshold, human-gated promotion, and automatic loading. This mirrors the architecture used by every mature finding-to-rule system studied, where the universal principle is: **automate the detection and drafting, but gate the codification on human judgment**."}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":3,"cache_creation_input_tokens":2658,"cache_read_input_tokens":7745,"output_tokens":3358,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":2658},"inference_geo":"","iterations":[],"speed":"standard"}},"requestId":"req_011CZEREoxWi3KVqkrEHwDfH","type":"assistant","uuid":"05d13811-3d96-427c-a51f-5b5ccf16a157","timestamp":"2026-03-20T13:35:36.006Z","userType":"external","entrypoint":"cli","cwd":"/Users/mohamedabdallah/Work/Wazir","sessionId":"0c5a9bfa-6816-42b1-a97d-37ae7dad26c5","version":"2.1.80","gitBranch":"feat/remaining-6-fixes","slug":"mighty-wondering-giraffe"}