code-ai-installer 2.5.0 → 4.0.0

package/domains/development/locales/en/.agents/skills/qa-test-integrity-audit/SKILL.md

@@ -0,0 +1,328 @@
+---
+name: qa-test-integrity-audit
+description: Static scanner for gaming patterns in tests — 9 categories (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement to mutation/property-based.
+type: discretionary
+domain: development
+owners:
+  - tester
+gates:
+  - TEST
+tech:
+  - eslint
+  - ruff
+topic:
+  - qa
+triggers:
+  - test integrity audit
+  - gaming patterns
+  - vacuous tests
+  - expect.anything
+  - snapshot without assertions
+  - skip only in commit
+  - deleted test in PR
+  - mock count audit
+  - sham tests
+related:
+  - qa-mutation-testing
+  - qa-property-based-testing
+  - tests-quality-review
+  - qa-test-plan
+  - tests-integrity-rules
+budget_lines: 340
+schema_version: 1
+---
+
+
+# Skill: QA Test Integrity Audit
+
+Static scanner for **gaming patterns** in tests — patterns through which tests formally pass but do not verify behavior. AI is especially prone to generate such tests when trying to "make a test green".
+
+Static defense (this skill) complements dynamic defense (`$qa-mutation-testing` + `$qa-property-based-testing`) — this is pillar 2 of Test Integrity Defense.
+
+**Sections:**
+1. [When to apply](#1-when)
+2. [9 detection patterns](#2-patterns)
+3. [ESLint plugin setup (JS/TS)](#3-eslint)
+4. [ruff plugin setup (Python)](#4-ruff)
+5. [Custom AST rules](#5-ast)
+6. [Pre-commit & CI](#6-ci)
+7. [Anti-patterns of the audit itself](#7-anti)
+8. [Output template](#8-output)
+9. [DoD](#9-dod)
+
+---
+
+## <a id="1-when"></a>1. When to apply
+
+**Automatically:**
+- Pre-commit hook on test files (.test.*, .spec.*, *_test.py)
+- CI check on PR (block merge on critical findings)
+
+**Manually:**
+- During test suite stabilization (regular audit every 2 weeks)
+- Before TEST signoff on critical modules (tier 1)
+- When reviewing PR with large test diff (>20% test changes)
+
+**Severity levels:**
+- 🔴 P0 — block merge automatically (e.g., deleted test without justification)
+- 🟠 P1 — flag in PR review (requires reviewer approve)
+- 🟡 P2 — info only (in audit report)
+
+---
+
+## <a id="2-patterns"></a>2. 9 detection patterns
+
+### 1. `expect.anything()` / `expect.any(X)` solo (🟠 P1)
+```ts
+// BAD
+expect(result).toEqual(expect.anything());
+// GOOD
+expect(result).toEqual({ status: "ok", id: expect.any(String) });
+```
+Detection: AST search `CallExpression[callee.name="anything"]` without siblings in matcher.
+
+### 2. `toBeDefined()` / `toBeTruthy()` without content (🟠 P1)
+```python
+# BAD
+assert user is not None
+# GOOD
+assert user is not None
+assert user.email == "x@y.com"
+assert user.role == "admin"
+```
+Detection: assertion call without following assertions on the same subject.
+
+### 3. Snapshot tests without assertions (🟠 P1)
+```ts
+// BAD — only assert is snapshot
+it("renders", () => expect(render(Foo)).toMatchSnapshot());
+// GOOD — snapshot + semantic
+it("renders", () => {
+  const html = render(Foo);
+  expect(html).toContain("Submit");
+  expect(html).toMatchSnapshot();
+});
+```
+Detection: `it()` with a single `toMatchSnapshot()` call.
+
+### 4. `.skip` / `.only` in commit (🔴 P0)
+```ts
+// BAD
+it.skip("missing test", () => {});
+describe.only("focused suite", () => {});
+```
+Detection: AST search `MemberExpression[property.name="skip"|"only"]` in test functions. ESLint rules `vitest/no-disabled-tests` + `vitest/no-focused-tests` cover this.
+
+### 5. `try/catch` swallows assertion (🔴 P0)
+```python
+# BAD — assert will fail, catch swallows it
+try:
+    assert user.balance > 0
+except Exception:
+    pass
+```
+Detection: AST search `TryStatement` where `body` contains assertion call AND `handler` is bare `except` / `catch (e) {}`.
+
+### 6. Deleted test without DELETED-WHY (🔴 P0)
+PR diff shows `- it("...")` or `- def test_...` block. Block merge if commit message lacks `DELETED-WHY: <reason>` field.
+Detection: pre-commit hook + gh check on PR diff parsing.
+
+### 7. Mock count increase without justification (🟠 P1)
+PR increases `vi.mock(...)` / `jest.mock(...)` / `unittest.mock.patch` calls by N. If N > 2 and no `MOCK-INCREASE-WHY:` in commit — flag.
+Detection: diff analysis script.
+
+### 8. Test without `expect()` / `assert` (🔴 P0)
+```ts
+// BAD — test "passes" being empty
+it("does something", () => {
+  doSomething();
+  // no assert
+});
+```
+Detection: AST search test functions without assertion calls anywhere in body.
+
+### 9. Tautology properties (🟠 P1 — echo from `$qa-property-based-testing`)
+```ts
+// BAD
+fc.assert(fc.property(fc.integer(), (x) => x === x));
+// GOOD
+fc.assert(fc.property(fc.integer(), (x) => abs(x) >= 0));
+```
+Detection: property body contains only `===` / `==` with itself OR `toBeDefined`-style.
+
+---
+
+## <a id="3-eslint"></a>3. ESLint plugin setup (JS/TS)
+
+**Install:**
+```bash
+npm i -D eslint @typescript-eslint/parser eslint-plugin-vitest eslint-plugin-jest
+```
+
+**Minimal config** (`.eslintrc.test.json` for test files):
+```json
+{
+  "overrides": [{
+    "files": ["**/*.{test,spec}.{ts,tsx}"],
+    "extends": ["plugin:vitest/recommended", "plugin:jest/recommended"],
+    "rules": {
+      "vitest/no-disabled-tests": "error",
+      "vitest/no-focused-tests": "error",
+      "vitest/expect-expect": "error",
+      "jest/no-commented-out-tests": "error",
+      "vitest/no-conditional-expect": "warn"
+    }
+  }]
+}
+```
+
+**Run:**
+```bash
+eslint --config .eslintrc.test.json "src/**/*.{test,spec}.{ts,tsx}"
+```
+
+Covered patterns: #1 (via `vitest/valid-expect`), #2 (partial), #4 (fully), #8 (via `vitest/expect-expect`).
+
+**Not covered out-of-box** (need custom AST rules or manual review): #3 (snapshot without semantic), #5 (try/catch swallows), #6 (deleted tests — it's git diff), #7 (mock count — diff), #9 (tautology).
+
+---
+
+## <a id="4-ruff"></a>4. ruff plugin setup (Python)
+
+**Install:**
+```bash
+pip install ruff
+```
+
+**Minimal config** (`pyproject.toml`):
+```toml
+[tool.ruff.lint]
+select = ["PT", "PL", "S101"]  # pytest-style + pylint + assert checks
+extend-select = ["PT001", "PT003", "PT006", "PT020"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**/*.py" = ["S101"]  # asserts OK in tests
+```
+
+**Run:**
+```bash
+ruff check tests/
+```
+
+Covered out-of-box: #4 (`PT006` — focused test detection), partially #8.
+
+**Not covered** (need custom plugin or manual): #1-3, #5-7, #9.
+
+For extended checks — pytest plugin `pytest-flakefinder` + custom AST plugin (see below).
+
+---
+
+## <a id="5-ast"></a>5. Custom AST rules
+
+For patterns #3 / #5 / #9 custom rules are needed. ESLint rule skeleton:
+
+```ts
+// .eslint-rules/no-snapshot-only.js
+module.exports = {
+  create(context) {
+    return {
+      "CallExpression[callee.name='it']"(node) {
+        const body = node.arguments[1]?.body?.body || [];
+        const asserts = body.filter(s =>
+          s.type === "ExpressionStatement" &&
+          s.expression.type === "CallExpression"
+        );
+        const snapshotsOnly = asserts.length === 1 &&
+          asserts[0].expression.callee?.property?.name === "toMatchSnapshot";
+        if (snapshotsOnly) {
+          context.report({ node, message: "Snapshot-only test — add semantic assertion" });
+        }
+      },
+    };
+  },
+};
+```
+
+Similar approach in Python — `ast.NodeVisitor` subclass, `visit_Try` finds assert inside bare-except handler, prints finding with line number. Place in `tools/test_audit/`.
+
+Custom rules location: `.eslint-rules/` (JS) or `tools/test_audit/` (Python). CI incrementally expands coverage as new gaming patterns are discovered.
+
+---
+
+## <a id="6-ci"></a>6. Pre-commit & CI
+
+**Pre-commit hook** (Husky / lefthook example):
+```yaml
+# .lefthook.yml
+pre-commit:
+  commands:
+    test-integrity:
+      glob: "*.{test,spec,_test}.*"
+      run: npm run lint:test {staged_files} && python tools/test_audit/run.py {staged_files}
+```
+
+**GitHub Actions check:**
+```yaml
+- name: Test integrity audit
+  run: |
+    npx eslint --config .eslintrc.test.json "src/**/*.{test,spec}.*"
+    ruff check tests/
+    python tools/test_audit/diff_check.py --base ${{ github.base_ref }}
+```
+
+Production-grade setup (`test-integrity-hooks-setup` skill DEV-092) — gh check block-merge, exemption mechanism, audit report aggregator. False-positive exemptions registry in V1 — committed to git.
+
+---
+
+## <a id="7-anti"></a>7. Anti-patterns of the audit itself
+
+- 🔴 **Over-strict without exemption** — rules without `// eslint-disable-next-line` escape fail legitimate cases (e.g., `expect.anything()` for dates when time is irrelevant). Always provide escape with require-comment-justification.
+- 🔴 **Audit-as-blocker without severity grading** — every P2 finding blocks merge — team will start ignoring it. P0 block, P1 review-required, P2 info-only.
+- 🟠 **False positives without feedback loop** — no channel "this is not gaming, legitimate case" — rules do not calibrate. Audit report must have `mark-as-legit` link.
+- 🟠 **Pattern coverage not updated** — gaming evolves (AI finds new workarounds), but rules are static. Quarterly review rules vs new patterns observed.
+- 🟡 **No documentation for each rule** — developer sees fail without explanation why. Each rule → markdown doc with bad/good example + reasoning.
+
+---
+
+## <a id="8-output"></a>8. Output template
+
+Tester / Reviewer includes in TEST / REV report:
+
+```
+### Test Integrity Audit
+- Files scanned: N
+- Patterns checked: 9
+- Findings: A P0 / B P1 / C P2
+- P0 (block):
+  - file:line — pattern X — [description]
+- P1 (review-required):
+  - file:line — pattern Y — [description]
+- P2 (info): N findings (see full report)
+- Mock count delta: +X (justified: yes/no)
+- Deleted tests: N (DELETED-WHY: yes/no per test)
+- Audit calibration: false positives reported this cycle: N
+```
+
+---
+
+## <a id="9-dod"></a>9. DoD
+
+- [ ] ESLint plugin (JS/TS) and/or ruff (Python) configured
+- [ ] Pre-commit hook runs audit on staged test files
+- [ ] CI check on PR with severity-graded findings
+- [ ] Custom AST rules for uncovered patterns (#3/#5/#9)
+- [ ] Output template included in TEST report template
+- [ ] Documentation for each custom rule (markdown with examples)
+- [ ] Feedback channel for false positives (mark-as-legit mechanism)
+
+---
+
+## See also
+
+- `$qa-mutation-testing` — dynamic test quality (pillar 1 — complement)
+- `$qa-property-based-testing` — input generation (pillar 1 — complement)
+- `$tests-quality-review` — Reviewer-side review (this skill feeds there)
+- `$qa-regression-baseline` — baseline tracking
+- `$qa-test-plan` — test plan defines coverage targets
+- `$tests-integrity-rules` — SFS-side rules (forbidden patterns, lint config skeleton)
+- `$test-integrity-hooks-setup` — pre-commit + gh check production setup

package/domains/development/locales/en/.agents/skills/qa-test-integrity-audit/agents/claude.json

@@ -0,0 +1,19 @@
+{
+  "name": "qa-test-integrity-audit",
+  "display_name": "QA Test Plan",
+  "description": "Static scanner for gaming patterns in tests — 9 categories (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement to mutation/property-based.",
+  "default_prompt": "Use $qa-test-plan when the task matches the \"QA Test Plan\" skill.",
+  "triggers": [
+    "qa-test-integrity-audit",
+    "qa test integrity audit",
+    "Create a test plan and coverage matrix based on PRD/UX — roles, critical user flows, UI states, negative cases, acceptance criteria"
+  ],
+  "capabilities": [
+    "qa",
+    "test",
+    "integrity",
+    "audit"
+  ],
+  "tools": [],
+  "implicit_invocation": true
+}

package/domains/development/locales/en/.agents/skills/qa-test-integrity-audit/agents/copilot.json

@@ -0,0 +1,19 @@
+{
+  "name": "qa-test-integrity-audit",
+  "display_name": "QA Test Plan",
+  "description": "Static scanner for gaming patterns in tests — 9 categories (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement to mutation/property-based.",
+  "default_prompt": "Use $qa-test-plan when the task matches the \"QA Test Plan\" skill.",
+  "triggers": [
+    "qa-test-integrity-audit",
+    "qa test integrity audit",
+    "Create a test plan and coverage matrix based on PRD/UX — roles, critical user flows, UI states, negative cases, acceptance criteria"
+  ],
+  "capabilities": [
+    "qa",
+    "test",
+    "integrity",
+    "audit"
+  ],
+  "tools": [],
+  "implicit_invocation": true
+}

package/domains/development/locales/en/.agents/skills/qa-test-integrity-audit/agents/gemini.json

@@ -0,0 +1,19 @@
+{
+  "name": "qa-test-integrity-audit",
+  "display_name": "QA Test Plan",
+  "description": "Static scanner for gaming patterns in tests — 9 categories (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement to mutation/property-based.",
+  "default_prompt": "Use $qa-test-plan when the task matches the \"QA Test Plan\" skill.",
+  "triggers": [
+    "qa-test-integrity-audit",
+    "qa test integrity audit",
+    "Create a test plan and coverage matrix based on PRD/UX — roles, critical user flows, UI states, negative cases, acceptance criteria"
+  ],
+  "capabilities": [
+    "qa",
+    "test",
+    "integrity",
+    "audit"
+  ],
+  "tools": [],
+  "implicit_invocation": true
+}

package/domains/development/locales/en/.agents/skills/qa-test-integrity-audit/agents/kimi.yaml

@@ -0,0 +1,15 @@
+name: "qa-test-integrity-audit"
+display_name: "QA Test Plan"
+description: "Static scanner for gaming patterns in tests — 9 categories (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement to mutation/property-based."
+default_prompt: "Use $qa-test-plan when the task matches the \"QA Test Plan\" skill."
+triggers:
+  - "qa-test-integrity-audit"
+  - "qa test integrity audit"
+  - "Create a test plan and coverage matrix based on PRD/UX — roles, critical user flows, UI states, negative cases, acceptance criteria"
+capabilities:
+  - "qa"
+  - "test"
+  - "integrity"
+  - "audit"
+tools: []
+implicit_invocation: true

package/domains/development/locales/en/.agents/skills/qa-test-integrity-audit/agents/openai.yaml

@@ -0,0 +1,8 @@
+interface:
+  display_name: "QA Test Plan"
+  short_description: "Create a test plan and coverage matrix based on PRD/UX — roles, critical user flows, UI states,…"
+  default_prompt: "Use $qa-test-plan when the task matches the \"QA Test Plan\" skill."
+dependencies:
+  tools: []
+policy:
+  allow_implicit_invocation: true

package/domains/development/locales/en/.agents/skills/qa-test-integrity-audit/agents/qwen.json

@@ -0,0 +1,19 @@
+{
+  "name": "qa-test-integrity-audit",
+  "display_name": "QA Test Plan",
+  "description": "Static scanner for gaming patterns in tests — 9 categories (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement to mutation/property-based.",
+  "default_prompt": "Use $qa-test-plan when the task matches the \"QA Test Plan\" skill.",
+  "triggers": [
+    "qa-test-integrity-audit",
+    "qa test integrity audit",
+    "Create a test plan and coverage matrix based on PRD/UX — roles, critical user flows, UI states, negative cases, acceptance criteria"
+  ],
+  "capabilities": [
+    "qa",
+    "test",
+    "integrity",
+    "audit"
+  ],
+  "tools": [],
+  "implicit_invocation": true
+}

package/domains/development/locales/en/.agents/skills/qa-test-integrity-audit/agents/skill.yaml

@@ -0,0 +1,23 @@
+version: 1
+name: "qa-test-integrity-audit"
+display_name: "QA Test Plan"
+description: "Static scanner for gaming patterns in tests — 9 categories (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement to mutation/property-based."
+default_prompt: "Use $qa-test-plan when the task matches the \"QA Test Plan\" skill."
+triggers:
+  - "qa-test-integrity-audit"
+  - "qa test integrity audit"
+  - "Create a test plan and coverage matrix based on PRD/UX — roles, critical user flows, UI states, negative cases, acceptance criteria"
+capabilities:
+  - "qa"
+  - "test"
+  - "integrity"
+  - "audit"
+tools: []
+invocation:
+  explicit: true
+  implicit: true
+localization:
+  default_locale: "en"
+  available_locales:
+    - "ru"
+    - "en"