code-ai-installer 3.0.0 → 4.0.1-a

package/domains/development/.agents/skills/qa-test-integrity-audit/SKILL.md

@@ -0,0 +1,328 @@
+---
+name: qa-test-integrity-audit
+description: Статический сканер на gaming patterns в тестах — 9 категорий (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement к mutation/property-based.
+type: discretionary
+domain: development
+owners:
+  - tester
+gates:
+  - TEST
+tech:
+  - eslint
+  - ruff
+topic:
+  - qa
+triggers:
+  - test integrity audit
+  - gaming patterns
+  - vacuous tests
+  - expect.anything
+  - снапшот без assertions
+  - skip only в коммите
+  - удалённый тест в PR
+  - mock count audit
+  - тесты-обманки
+related:
+  - qa-mutation-testing
+  - qa-property-based-testing
+  - tests-quality-review
+  - qa-test-plan
+  - tests-integrity-rules
+budget_lines: 340
+schema_version: 1
+---
+
+
+# Skill: QA Test Integrity Audit
+
+Статический сканер на **gaming patterns** в тестах — паттерны через которые тесты формально проходят, но не testируют поведение. AI особенно склонен генерировать такие тесты при попытке "сделать тест зелёным".
+
+Static defense (этот скилл) дополняет dynamic defense (`$qa-mutation-testing` + `$qa-property-based-testing`) — это pillar 2 Test Integrity Defense.
+
+**Разделы:**
+1. [Когда применять](#1-когда)
+2. [9 detection patterns](#2-patterns)
+3. [ESLint plugin setup (JS/TS)](#3-eslint)
+4. [ruff plugin setup (Python)](#4-ruff)
+5. [Custom AST rules](#5-ast)
+6. [Pre-commit & CI](#6-ci)
+7. [Anti-patterns самого audit'а](#7-anti)
+8. [Output template](#8-output)
+9. [DoD](#9-dod)
+
+---
+
+## <a id="1-когда"></a>1. Когда применять
+
+**Автоматически:**
+- Pre-commit hook на test files (.test.*, .spec.*, *_test.py)
+- CI check на PR (block merge при critical findings)
+
+**Вручную:**
+- При стабилизации test suite (regular audit раз в 2 недели)
+- Перед TEST signoff на критичных модулях (tier 1)
+- При ревью PR с большим diff в тестах (>20% test changes)
+
+**Severity levels:**
+- 🔴 P0 — block merge automatically (например, удалённый тест без обоснования)
+- 🟠 P1 — flag в PR review (требует ревьюера на approve)
+- 🟡 P2 — info только (в audit report)
+
+---
+
+## <a id="2-patterns"></a>2. 9 detection patterns
+
+### 1. `expect.anything()` / `expect.any(X)` соло (🟠 P1)
+```ts
+// BAD
+expect(result).toEqual(expect.anything());
+// GOOD
+expect(result).toEqual({ status: "ok", id: expect.any(String) });
+```
+Detection: AST search `CallExpression[callee.name="anything"]` без siblings в matcher.
+
+### 2. `toBeDefined()` / `toBeTruthy()` без content (🟠 P1)
+```python
+# BAD
+assert user is not None
+# GOOD
+assert user is not None
+assert user.email == "x@y.com"
+assert user.role == "admin"
+```
+Detection: assertion вызов без следующих assertions на тот же subject.
+
+### 3. Snapshot тесты без assertions (🟠 P1)
+```ts
+// BAD — единственный assert это snapshot
+it("renders", () => expect(render(Foo)).toMatchSnapshot());
+// GOOD — snapshot + semantic
+it("renders", () => {
+  const html = render(Foo);
+  expect(html).toContain("Submit");
+  expect(html).toMatchSnapshot();
+});
+```
+Detection: `it()` с единственным `toMatchSnapshot()` вызовом.
+
+### 4. `.skip` / `.only` в commit (🔴 P0)
+```ts
+// BAD
+it.skip("missing test", () => {});
+describe.only("focused suite", () => {});
+```
+Detection: AST search `MemberExpression[property.name="skip"|"only"]` в test functions. ESLint rule `vitest/no-disabled-tests` + `vitest/no-focused-tests` покрывают.
+
+### 5. `try/catch` глотает assertion (🔴 P0)
+```python
+# BAD — assert упадёт, catch скушает
+try:
+    assert user.balance > 0
+except Exception:
+    pass
+```
+Detection: AST search `TryStatement` где `body` contains assertion call AND `handler` is bare `except` / `catch (e) {}`.
+
+### 6. Удалённый тест без DELETED-WHY (🔴 P0)
+PR diff показывает `- it("...")` или `- def test_...` блок. Block merge если в commit message нет `DELETED-WHY: <причина>` поля.
+Detection: pre-commit hook + gh check на PR diff parsing.
+
+### 7. Mock count увеличение без обоснования (🟠 P1)
+PR увеличивает `vi.mock(...)` / `jest.mock(...)` / `unittest.mock.patch` calls на N штук. Если N > 2 и нет `MOCK-INCREASE-WHY:` в commit — flag.
+Detection: diff analysis script.
+
+### 8. Test без `expect()` / `assert` (🔴 P0)
+```ts
+// BAD — тест "проходит" пустым
+it("does something", () => {
+  doSomething();
+  // никакого assert
+});
+```
+Detection: AST search test functions без assertion calls anywhere in body.
+
+### 9. Tautology properties (🟠 P1 — echo из `$qa-property-based-testing`)
+```ts
+// BAD
+fc.assert(fc.property(fc.integer(), (x) => x === x));
+// GOOD
+fc.assert(fc.property(fc.integer(), (x) => abs(x) >= 0));
+```
+Detection: property body содержит только `===` / `==` сам с собой OR `toBeDefined`-style.
+
+---
+
+## <a id="3-eslint"></a>3. ESLint plugin setup (JS/TS)
+
+**Установка:**
+```bash
+npm i -D eslint @typescript-eslint/parser eslint-plugin-vitest eslint-plugin-jest
+```
+
+**Минимальный конфиг** (`.eslintrc.test.json` для test files):
+```json
+{
+  "overrides": [{
+    "files": ["**/*.{test,spec}.{ts,tsx}"],
+    "extends": ["plugin:vitest/recommended", "plugin:jest/recommended"],
+    "rules": {
+      "vitest/no-disabled-tests": "error",
+      "vitest/no-focused-tests": "error",
+      "vitest/expect-expect": "error",
+      "jest/no-commented-out-tests": "error",
+      "vitest/no-conditional-expect": "warn"
+    }
+  }]
+}
+```
+
+**Запуск:**
+```bash
+eslint --config .eslintrc.test.json "src/**/*.{test,spec}.{ts,tsx}"
+```
+
+Covered patterns: #1 (через `vitest/valid-expect`), #2 (partial), #4 (полностью), #8 (через `vitest/expect-expect`).
+
+**Не covered out-of-box** (нужны custom AST rules или manual review): #3 (snapshot без semantic), #5 (try/catch swallows), #6 (deleted tests — это git diff), #7 (mock count — diff), #9 (tautology).
+
+---
+
+## <a id="4-ruff"></a>4. ruff plugin setup (Python)
+
+**Установка:**
+```bash
+pip install ruff
+```
+
+**Минимальный конфиг** (`pyproject.toml`):
+```toml
+[tool.ruff.lint]
+select = ["PT", "PL", "S101"]  # pytest-style + pylint + assert checks
+extend-select = ["PT001", "PT003", "PT006", "PT020"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**/*.py" = ["S101"]  # asserts OK в тестах
+```
+
+**Запуск:**
+```bash
+ruff check tests/
+```
+
+Covered out-of-box: #4 (`PT006` — focused test detection), частично #8.
+
+**Не covered** (нужны custom plugin или manual): #1-3, #5-7, #9.
+
+Для расширенных проверок — pytest plugin `pytest-flakefinder` + кастомный AST plugin (см. ниже).
+
+---
+
+## <a id="5-ast"></a>5. Custom AST rules
+
+Для паттернов #3 / #5 / #9 нужны custom rules. Скелет ESLint rule:
+
+```ts
+// .eslint-rules/no-snapshot-only.js
+module.exports = {
+  create(context) {
+    return {
+      "CallExpression[callee.name='it']"(node) {
+        const body = node.arguments[1]?.body?.body || [];
+        const asserts = body.filter(s =>
+          s.type === "ExpressionStatement" &&
+          s.expression.type === "CallExpression"
+        );
+        const snapshotsOnly = asserts.length === 1 &&
+          asserts[0].expression.callee?.property?.name === "toMatchSnapshot";
+        if (snapshotsOnly) {
+          context.report({ node, message: "Snapshot-only test — add semantic assertion" });
+        }
+      },
+    };
+  },
+};
+```
+
+Аналогичный подход в Python — `ast.NodeVisitor` подкласс, `visit_Try` находит assert внутри bare-except handler, печатает finding с line number. Размещение в `tools/test_audit/`.
+
+Размещение custom rules: `.eslint-rules/` (JS) или `tools/test_audit/` (Python). CI инкрементально расширяет coverage по мере обнаружения новых gaming patterns.
+
+---
+
+## <a id="6-ci"></a>6. Pre-commit & CI
+
+**Pre-commit hook** (Husky / lefthook example):
+```yaml
+# .lefthook.yml
+pre-commit:
+  commands:
+    test-integrity:
+      glob: "*.{test,spec,_test}.*"
+      run: npm run lint:test {staged_files} && python tools/test_audit/run.py {staged_files}
+```
+
+**GitHub Actions check:**
+```yaml
+- name: Test integrity audit
+  run: |
+    npx eslint --config .eslintrc.test.json "src/**/*.{test,spec}.*"
+    ruff check tests/
+    python tools/test_audit/diff_check.py --base ${{ github.base_ref }}
+```
+
+Production-grade setup (`test-integrity-hooks-setup` skill DEV-092) — gh check block-merge, exemption mechanism, audit report aggregator. False-positive exemptions registry в V1 — коммитим в git.
+
+---
+
+## <a id="7-anti"></a>7. Anti-patterns самого audit'а
+
+- 🔴 **Over-strict без exemption** — rules без `// eslint-disable-next-line` escape валят легитимные кейсы (например `expect.anything()` для дат когда время не важно). Always provide escape с require-comment-justification.
+- 🔴 **Audit-as-blocker без severity градации** — каждый P2 finding блокирует merge — команда начнёт игнорировать. P0 block, P1 review-required, P2 info-only.
+- 🟠 **False positives без feedback loop** — нет канала "это не gaming, легитимный кейс" — rules не calibrate. Audit report должен иметь `mark-as-legit` link.
+- 🟠 **Pattern coverage не updated** — gaming evolves (AI находит новые workarounds), а rules statiс. Quarterly review rules vs new patterns observed.
+- 🟡 **No documentation для каждого rule** — разработчик видит fail без объяснения почему. Each rule → markdown doc с примером bad/good + reasoning.
+
+---
+
+## <a id="8-output"></a>8. Output template
+
+Tester / Reviewer включает в TEST / REV report:
+
+```
+### Test Integrity Audit
+- Files scanned: N
+- Patterns checked: 9
+- Findings: A P0 / B P1 / C P2
+- P0 (block):
+  - file:line — pattern X — [description]
+- P1 (review-required):
+  - file:line — pattern Y — [description]
+- P2 (info): N findings (см. полный отчёт)
+- Mock count delta: +X (justified: yes/no)
+- Deleted tests: N (DELETED-WHY: yes/no per test)
+- Audit calibration: false positives reported this cycle: N
+```
+
+---
+
+## <a id="9-dod"></a>9. DoD
+
+- [ ] ESLint plugin (JS/TS) и/или ruff (Python) сконфигурированы
+- [ ] Pre-commit hook запускает audit на staged test files
+- [ ] CI check на PR с severity-graded findings
+- [ ] Custom AST rules для непокрытых паттернов (#3/#5/#9)
+- [ ] Output template включён в TEST report шаблон
+- [ ] Documentation для каждого custom rule (markdown с примерами)
+- [ ] Feedback channel для false positives (mark-as-legit mechanism)
+
+---
+
+## См. также
+
+- `$qa-mutation-testing` — dynamic test quality (pillar 1 — complement)
+- `$qa-property-based-testing` — input generation (pillar 1 — complement)
+- `$tests-quality-review` — Reviewer-side review (этот скилл feeds туда)
+- `$qa-regression-baseline` — baseline tracking
+- `$qa-test-plan` — test plan определяет coverage цели
+- `$tests-integrity-rules` — SFS-side правила (forbidden patterns, lint config skeleton)
+- `$test-integrity-hooks-setup` — pre-commit + gh check production setup

package/domains/development/.agents/skills/qa-test-integrity-audit/agents/claude.json

@@ -0,0 +1,20 @@
+{
+  "name": "qa-test-integrity-audit",
+  "display_name": "QA: test integrity audit",
+  "description": "Статический сканер на gaming patterns в тестах — 9 категорий (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement к mutation/property-based.",
+  "default_prompt": "Используй $qa-test-plan, когда задача относится к навыку \"QA: test integrity audit\".",
+  "triggers": [
+    "qa-test-integrity-audit",
+    "qa test integrity audit",
+    "QA: test integrity audit",
+    "Составить тест-план и матрицу покрытия по PRD/UX — роли, критичные user flows, состояния UI, негативные кейсы, критерии приёмки"
+  ],
+  "capabilities": [
+    "qa",
+    "test",
+    "integrity",
+    "audit"
+  ],
+  "tools": [],
+  "implicit_invocation": true
+}

package/domains/development/.agents/skills/qa-test-integrity-audit/agents/copilot.json

@@ -0,0 +1,20 @@
+{
+  "name": "qa-test-integrity-audit",
+  "display_name": "QA: test integrity audit",
+  "description": "Статический сканер на gaming patterns в тестах — 9 категорий (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement к mutation/property-based.",
+  "default_prompt": "Используй $qa-test-plan, когда задача относится к навыку \"QA: test integrity audit\".",
+  "triggers": [
+    "qa-test-integrity-audit",
+    "qa test integrity audit",
+    "QA: test integrity audit",
+    "Составить тест-план и матрицу покрытия по PRD/UX — роли, критичные user flows, состояния UI, негативные кейсы, критерии приёмки"
+  ],
+  "capabilities": [
+    "qa",
+    "test",
+    "integrity",
+    "audit"
+  ],
+  "tools": [],
+  "implicit_invocation": true
+}

package/domains/development/.agents/skills/qa-test-integrity-audit/agents/gemini.json

@@ -0,0 +1,20 @@
+{
+  "name": "qa-test-integrity-audit",
+  "display_name": "QA: test integrity audit",
+  "description": "Статический сканер на gaming patterns в тестах — 9 категорий (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement к mutation/property-based.",
+  "default_prompt": "Используй $qa-test-plan, когда задача относится к навыку \"QA: test integrity audit\".",
+  "triggers": [
+    "qa-test-integrity-audit",
+    "qa test integrity audit",
+    "QA: test integrity audit",
+    "Составить тест-план и матрицу покрытия по PRD/UX — роли, критичные user flows, состояния UI, негативные кейсы, критерии приёмки"
+  ],
+  "capabilities": [
+    "qa",
+    "test",
+    "integrity",
+    "audit"
+  ],
+  "tools": [],
+  "implicit_invocation": true
+}

package/domains/development/.agents/skills/qa-test-integrity-audit/agents/kimi.yaml

@@ -0,0 +1,16 @@
+name: "qa-test-integrity-audit"
+display_name: "QA: test integrity audit"
+description: "Статический сканер на gaming patterns в тестах — 9 категорий (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement к mutation/property-based."
+default_prompt: "Используй $qa-test-plan, когда задача относится к навыку \"QA: test integrity audit\"."
+triggers:
+  - "qa-test-integrity-audit"
+  - "qa test integrity audit"
+  - "QA: test integrity audit"
+  - "Составить тест-план и матрицу покрытия по PRD/UX — роли, критичные user flows, состояния UI, негативные кейсы, критерии приёмки"
+capabilities:
+  - "qa"
+  - "test"
+  - "integrity"
+  - "audit"
+tools: []
+implicit_invocation: true

package/domains/development/.agents/skills/qa-test-integrity-audit/agents/openai.yaml

@@ -0,0 +1,8 @@
+interface:
+  display_name: "QA: test integrity audit"
+  short_description: "Составить тест-план и матрицу покрытия по PRD/UX — роли, критичные user flows, состояния UI, не…"
+  default_prompt: "Используй $qa-test-plan, когда задача относится к навыку \"QA: test integrity audit\"."
+dependencies:
+  tools: []
+policy:
+  allow_implicit_invocation: true

package/domains/development/.agents/skills/qa-test-integrity-audit/agents/qwen.json

@@ -0,0 +1,20 @@
+{
+  "name": "qa-test-integrity-audit",
+  "display_name": "QA: test integrity audit",
+  "description": "Статический сканер на gaming patterns в тестах — 9 категорий (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement к mutation/property-based.",
+  "default_prompt": "Используй $qa-test-plan, когда задача относится к навыку \"QA: test integrity audit\".",
+  "triggers": [
+    "qa-test-integrity-audit",
+    "qa test integrity audit",
+    "QA: test integrity audit",
+    "Составить тест-план и матрицу покрытия по PRD/UX — роли, критичные user flows, состояния UI, негативные кейсы, критерии приёмки"
+  ],
+  "capabilities": [
+    "qa",
+    "test",
+    "integrity",
+    "audit"
+  ],
+  "tools": [],
+  "implicit_invocation": true
+}

package/domains/development/.agents/skills/qa-test-integrity-audit/agents/skill.yaml

@@ -0,0 +1,24 @@
+version: 1
+name: "qa-test-integrity-audit"
+display_name: "QA: test integrity audit"
+description: "Статический сканер на gaming patterns в тестах — 9 категорий (expect.anything, weak assertions, snapshot drift, .skip/.only, try/catch swallows, deleted tests, mock count, no expect, tautology properties). ESLint + ruff plugins, custom AST rules, pre-commit hook. Static defense — complement к mutation/property-based."
+default_prompt: "Используй $qa-test-plan, когда задача относится к навыку \"QA: test integrity audit\"."
+triggers:
+  - "qa-test-integrity-audit"
+  - "qa test integrity audit"
+  - "QA: test integrity audit"
+  - "Составить тест-план и матрицу покрытия по PRD/UX — роли, критичные user flows, состояния UI, негативные кейсы, критерии приёмки"
+capabilities:
+  - "qa"
+  - "test"
+  - "integrity"
+  - "audit"
+tools: []
+invocation:
+  explicit: true
+  implicit: true
+localization:
+  default_locale: "ru"
+  available_locales:
+    - "ru"
+    - "en"