@cbartens/bkit 5.2.2 → 5.2.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.bkit/README.md +4 -2
- package/.bkit/core/.agents/context.md +2 -2
- package/.bkit/core/.agents/plugins/bkit/hooks/bkit-guardrails.cjs +7 -1
- package/.bkit/core/.bkit/README.md +3 -1
- package/.bkit/core/.bkit/comparison.md +114 -45
- package/.bkit/core/.bkit/install.md +6 -6
- package/.bkit/core/.bkit/jsonl.md +3 -3
- package/.bkit/core/.bkit/prompts.md +6 -5
- package/.bkit/core/.bkit/providers.md +5 -4
- package/.bkit/core/.bkit/templates.md +30 -11
- package/.bkit/core/.bkit/workflows.md +14 -7
- package/.bkit/core/.codex/hooks/bkit-guardrails.cjs +5 -1
- package/.bkit/core/.cursor/hooks/bkit-guardrails.cjs +8 -2
- package/.bkit/core/docs/templates/bkit-audit-accessibility.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-adoption.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-agents.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-aio-seo.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-analytics.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-annotations.md +15 -4
- package/.bkit/core/docs/templates/bkit-audit-app-performance.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-app-store.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-architecture.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-ci-performance.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-config.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-cost.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-coverage.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-database.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-dead-code.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-deferrals.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-dependencies.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-documentation.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-duplication.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-failures.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-flakes.md +15 -4
- package/.bkit/core/docs/templates/bkit-audit-gates.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-issues.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-llm.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-logs.md +25 -13
- package/.bkit/core/docs/templates/bkit-audit-models.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-privacy.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-prompts.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-red-team.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-reliability.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-repo.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-security.md +15 -4
- package/.bkit/core/docs/templates/bkit-audit-slop.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-testing.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-usability.md +13 -2
- package/.bkit/core/docs/templates/bkit-plan.md +1 -1
- package/.bkit/core/engine/checks/tips/catalogue.ts +10 -0
- package/.bkit/core/engine/commands/install/check.ts +3 -11
- package/.bkit/core/engine/commands/install/doctor.ts +2 -3
- package/.bkit/core/engine/commands/install/uninstall.ts +60 -44
- package/.bkit/core/engine/commands/install/upgrade.ts +13 -11
- package/.bkit/core/engine/commands/workflow/branches.ts +72 -9
- package/.bkit/core/engine/commands/workflow/repo.ts +16 -2
- package/.bkit/core/engine/commands/workflow/workflows.ts +4 -3
- package/.bkit/core/engine/install/hosts.ts +14 -4
- package/.bkit/core/engine/ledgers/telemetry.ts +3 -4
- package/.bkit/core/engine/shared/bkit-latest.ts +164 -0
- package/.bkit/core/engine/shared/timeouts.ts +6 -0
- package/.bkit/core/hooks/guardrails-read.js +4 -9
- package/.bkit/core/hooks/guardrails-tiers.js +312 -28
- package/.bkit/core/prompts/bkit-audit-generate.md +21 -5
- package/.bkit/core/workflows/bkit-build.md +1 -1
- package/.bkit/core/workflows/bkit-debug.md +1 -1
- package/.bkit/core/workflows/bkit-design.md +1 -1
- package/.bkit/core/workflows/bkit-plan.md +1 -1
- package/.bkit/core/workflows/bkit-ship.md +1 -1
- package/.bkit/shared/.claude/settings.json +19 -0
- package/.bkit/shared/.opencode/opencode.json +21 -1
- package/.bkit/shared/docs/deployments.md +14 -0
- package/.bkit/templates/.github/workflows/bkit-ci.yml +9 -0
- package/.bkit/templates/.husky/pre-push +7 -0
- package/.bkit/templates/context/bkit-cloudrun.md +28 -19
- package/.bkit/templates/context/bkit-expo.md +30 -13
- package/.bkit/templates/context/bkit-inngest.md +54 -17
- package/.bkit/templates/context/bkit-resend.md +36 -8
- package/.bkit/templates/context/bkit-sentry.md +11 -4
- package/.bkit/templates/context/bkit-stripe.md +39 -22
- package/.bkit/templates/context/bkit-supabase.md +45 -20
- package/.bkit/templates/context/bkit-vercel.md +45 -16
- package/.bkit/templates/scripts/ci/accepted-advisories.json +7 -0
- package/.bkit/templates/scripts/ci/check-dependency-audit.mjs +200 -0
- package/.bkit/templates/scripts/ci/check-dependency-audit.test.mjs +127 -0
- package/.bkit/templates/scripts/cleanup/bkit-tmp-prune.mjs +115 -0
- package/CHANGELOG.md +56 -0
- package/README.md +76 -20
- package/dist/checks/tips/catalogue.js +9 -0
- package/dist/commands/install/check.js +3 -15
- package/dist/commands/install/doctor.js +2 -1
- package/dist/commands/install/uninstall.js +52 -45
- package/dist/commands/install/upgrade.js +13 -11
- package/dist/commands/workflow/branches.js +64 -10
- package/dist/commands/workflow/repo.js +14 -2
- package/dist/commands/workflow/workflows.js +4 -3
- package/dist/install/hosts.js +11 -4
- package/dist/shared/bkit-latest.js +150 -0
- package/dist/shared/timeouts.js +6 -0
- package/package.json +1 -1
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -86,10 +86,13 @@ Batch verdict, in `Summary`, not per PR. The landing push's CI is the next step'
|
|
|
86
86
|
|
|
87
87
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
88
88
|
|
|
89
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
90
|
+
|
|
89
91
|
#### Core
|
|
90
92
|
|
|
91
93
|
Always verify.
|
|
92
94
|
|
|
95
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
93
96
|
- [ ] **Git Host:** Steps 2, 3, 11, 12 — open PRs by head-ref prefix, CI runs, behind-count, e.g. `gh`, GitLab API. Fallback: none; PR steps `Not Checked`, batch verdict `FAIL`.
|
|
94
97
|
- [ ] **Package Manager:** Steps 5, 8–10 — install, lockfile, registry metadata, e.g. pnpm, npm. Fallback: none; `Batch` and `Verification` `Not Checked`, batch verdict `FAIL`.
|
|
95
98
|
- [ ] **Clean Worktree:** Steps 8–10 — detached worktree off the integration branch, removed before the report finishes. Fallback: none; do not build the batch in the main checkout.
|
|
@@ -112,7 +115,7 @@ Verify only when applicable.
|
|
|
112
115
|
|
|
113
116
|
### Probe Steps
|
|
114
117
|
|
|
115
|
-
- [ ] **Step 1 — Previous Run:**
|
|
118
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
116
119
|
- [ ] **Step 2 — Inventory:** From previous report, list deferred PRs and why. Deferred major whose blocker cleared → first candidate. Select open PRs by head-ref prefix from updater config (`branchPrefix`, default `renovate/`; Dependabot `dependabot/`), never title or author. Token-authored PR on prefix = in scope; record author as evidence, not selector. Per PR: number, title, head ref, head SHA, mergeable, merge state, files, labels, commits behind integration branch, latest CI run (finished only; in-progress ≠ result). **Done when:** `Inventory` names prefix and config source, lists every open PR on prefix, or `none`.
|
|
117
120
|
- [ ] **Step 3 — Bot Health:** No dependency updater config = CRITICAL. Examples: Renovate (`.github/renovate.json` or `.github/renovate.json5`; rules example `.bkit/templates/.github/renovate.json5`, self-hosted runner example `.bkit/templates/.github/workflows/bkit-renovate.yml`) or Dependabot (`.github/dependabot.yml`; example `.bkit/templates/.github/dependabot.yml`). See `docs/deployments.md` § Dependency updaters. Both configs = finding; keep one. Bot dashboard: gather-only (bot rewrites). Record blocked entries and will-not-auto-rebase notices → stale until person acts. Bot reads presets from default branch, not integration. Fence / group on integration branch inactive until release carries it. Base-config edits on integration apply only if bot merges base-branch config — option read from default branch. PR proposes what integration config blocks → read default-branch copy before debugging rule; promotion-lag when branches differ = signal. Validate configs as default branch has them. Dashboard + logs: config errors, PRs held by concurrency / hourly limits, awaiting approval, open config-migration PRs. Broken config ≈ quiet week. **Done when:** `Bot Health` holds blocked PRs, config read branch, validator result, held / awaiting-approval items.
|
|
118
121
|
- [ ] **Step 4 — Supply Chain:** Read settings as default branch has them. Record each vs bot best-practice preset and package manager supply-chain page. Bot: minimum release age >= three days (e.g. Renovate `minimumReleaseAge` / `security:minimumReleaseAgeNpm` + `internalChecksFilter: "strict"`; Dependabot `cooldown`); actions / images digest-pinned; abandonment flagging (e.g. Renovate abandonments preset); config-migration PRs enabled, where bot offers them. Package manager, where offered (e.g. pnpm): trust policy no-downgrade, exotic subdeps blocked, explicit install-script allowlist, strict dependency builds; none offered → `N/A` with evidence. Release-age 0 on package manager = recorded trade-off where docs say so; re-deciding = person (`DECISION`). Gap: bot release age does not apply to lockfile-maintenance PRs (package manager makes changes; gate off → no cooldown) → batch with no cooldown; `Cooldown` stands in; `Screen` first. Missing / weaker setting → `Recommendations` row + line to add. **Done when:** `Supply Chain` row per setting: value · standard · verdict.
|
|
@@ -246,6 +249,13 @@ PENDING
|
|
|
246
249
|
|
|
247
250
|
PENDING
|
|
248
251
|
|
|
252
|
+
### Suggested Issues
|
|
253
|
+
|
|
254
|
+
<!-- prettier-ignore -->
|
|
255
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
256
|
+
| --- | --- | --- | --- | --- | --- |
|
|
257
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
258
|
+
|
|
249
259
|
### Process Improvements
|
|
250
260
|
|
|
251
261
|
PENDING
|
|
@@ -259,6 +269,7 @@ PENDING
|
|
|
259
269
|
- [ ] **Series:** security-fix latency and end-of-life days beside previous report's numbers, or marked baseline.
|
|
260
270
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
261
271
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
272
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
262
273
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
263
274
|
|
|
264
275
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -73,10 +73,13 @@ Docs are the record agents work from: memory points at them, plans cite them, co
|
|
|
73
73
|
|
|
74
74
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
75
75
|
|
|
76
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
77
|
+
|
|
76
78
|
#### Core
|
|
77
79
|
|
|
78
80
|
Always verify.
|
|
79
81
|
|
|
82
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
80
83
|
- [ ] **Repo Read:** Steps 2–13 — live docs, instruction files, manifests, CI workflow. Fallback: none; unreadable tree = finding.
|
|
81
84
|
- [ ] **Web:** Step 14 — fetch anchor standards. Fallback: versions recorded as unread.
|
|
82
85
|
|
|
@@ -100,7 +103,7 @@ Verify only when applicable.
|
|
|
100
103
|
|
|
101
104
|
### Probe Steps
|
|
102
105
|
|
|
103
|
-
- [ ] **Step 1 — Previous Run:**
|
|
106
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
104
107
|
- [ ] **Step 2 — Scope:** No live docs → one `N/A` row with evidence and STOP. Else every live doc. Rank by drift: per live doc, commits since that doc last changed to repo paths it cites. Record ranking. Start from up-to-date default branch with clean `git status`. Historical records are citers: pointer leads nowhere = finding, fix at target. Stale fact inside them = history, not defect. **Done when:** every live doc and drift ranking in `Scope`.
|
|
105
108
|
- [ ] **Step 3 — Catalogue:** `.agents/docs.md`: every file on disk under `docs/` outside `plans/`, `archive/`, and `audits/` has a row, and every row's file exists; no doc listed twice (two docs on one topic = bug, including one doc in both tables with different reading columns); each row's one-line scope still matches the doc. Root `README.md` = working entry point: what the project is, links setup doc, architecture doc, and `.agents/docs.md`, and how to report a security issue. Missing community-health file with no README substitute = finding. **Done when:** `Catalogue` names every mismatch, and every clean check.
|
|
106
109
|
- [ ] **Step 4 — Mechanical:** Run prose lint, format check, and managed-block check. Then:
|
|
@@ -258,6 +261,13 @@ PENDING
|
|
|
258
261
|
|
|
259
262
|
PENDING
|
|
260
263
|
|
|
264
|
+
### Suggested Issues
|
|
265
|
+
|
|
266
|
+
<!-- prettier-ignore -->
|
|
267
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
268
|
+
| --- | --- | --- | --- | --- | --- |
|
|
269
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
270
|
+
|
|
261
271
|
### Process Improvements
|
|
262
272
|
|
|
263
273
|
PENDING
|
|
@@ -271,6 +281,7 @@ PENDING
|
|
|
271
281
|
- [ ] **Series:** load size, and setup-walk times when the walk ran, beside previous report's numbers, or marked baseline.
|
|
272
282
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
273
283
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
284
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
274
285
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
275
286
|
|
|
276
287
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -90,10 +90,13 @@ Carry `COINCIDENTAL` and `RULE-OF-THREE-PENDING` forward in `Carried Forward`
|
|
|
90
90
|
|
|
91
91
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
92
92
|
|
|
93
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
94
|
+
|
|
93
95
|
#### Core
|
|
94
96
|
|
|
95
97
|
Always verify.
|
|
96
98
|
|
|
99
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
97
100
|
- [ ] **Clean Tree:** Steps 2–13 — clean `git status` at recorded commit; findings only mean something against a known tree. Fallback: detached worktree off default branch.
|
|
98
101
|
- [ ] **Text Search:** Steps 3, 5, 7–10 — inline expressions, literals, env names, non-code files, e.g. ripgrep, `git grep`. Fallback: `grep -rn`.
|
|
99
102
|
- [ ] **Structural Search:** Steps 3, 12 — syntax patterns that skip comments / strings and catch reordered copies, e.g. ast-grep, Semgrep; ad hoc run, record resolved version. Fallback: text search; reformatted copies `Not Checked`.
|
|
@@ -120,7 +123,7 @@ Verify only when applicable.
|
|
|
120
123
|
|
|
121
124
|
### Probe Steps
|
|
122
125
|
|
|
123
|
-
- [ ] **Step 1 — Previous Run:**
|
|
126
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
124
127
|
- [ ] **Step 2 — Detect:** List apps, packages, platforms; shared homes (utility / owner modules); capability register; duplication gates enforced and what each catches (e.g. identical-signature gate → identical copies only). **Done when:** surface map and existing gates in `Checked`, or `N/A` and STOP.
|
|
125
128
|
- [ ] **Step 3 — Rule Altitude:** Hunt one conceptual operation inlined at N sites instead of called from one function, copies differing on load-bearing details: fallback chains (e.g. JS `a ?? b ?? c`, divergent terminal fallback or order); null vs falsy (e.g. JS `??` vs `||`, with / without trim, optional-chain guards); formatting (date / time / timezone, currency, name / initials, address); predicates ("is active / deleted / owner?" filters); defaults (same enum / config default hardcoded in several readers). Structural rules where possible, text search where not; keep rules under `tmp/`, paste each into report for next run. Subtraction per shared helper: references on helper (routed set) minus sites matching inline pattern (inline set) → inline-not-routed = unmigrated site. Counts before reference-graph change (e.g. project references added) = stale → re-take, never carry forward. **Done when:** `Inline Rules` row per rule: helper · routed · inline · unmigrated · verdict, with its search rule.
|
|
126
129
|
- [ ] **Step 4 — Clones:** Clone detector over source roots, machine-readable output, three passes: Type-1 exact; Type-2 renamed (ignore identifiers / literals); Type-3 near-miss (e.g. similarity 0.85, syntax-tree mode). Duplication % per package as series vs previous report. Every hit = candidate for `Verify`, not finding. Multiple platforms → counterpart listing (functions with / without counterpart per side). Growth: exact pass baselined at previous report's commit → clones new since vs consolidated since, blame naming introducer; arrival faster than consolidation = finding. **Done when:** per-package %, growth, counterpart list in `Clones`.
|
|
@@ -231,6 +234,13 @@ PENDING
|
|
|
231
234
|
|
|
232
235
|
PENDING
|
|
233
236
|
|
|
237
|
+
### Suggested Issues
|
|
238
|
+
|
|
239
|
+
<!-- prettier-ignore -->
|
|
240
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
241
|
+
| --- | --- | --- | --- | --- | --- |
|
|
242
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
243
|
+
|
|
234
244
|
### Process Improvements
|
|
235
245
|
|
|
236
246
|
PENDING
|
|
@@ -244,6 +254,7 @@ PENDING
|
|
|
244
254
|
- [ ] **Series:** per-package duplication % and clone growth beside previous report's numbers, or marked baseline.
|
|
245
255
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
246
256
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
257
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
247
258
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
248
259
|
|
|
249
260
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -88,10 +88,13 @@ Row text awaiting confirmation = UNCONFIRMED until reproduced or read end to end
|
|
|
88
88
|
|
|
89
89
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
90
90
|
|
|
91
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
92
|
+
|
|
91
93
|
#### Core
|
|
92
94
|
|
|
93
95
|
Always verify.
|
|
94
96
|
|
|
97
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
95
98
|
- [ ] **Ledger:** Steps 3–7 — `docs/failures.jsonl` and `.bkit/jsonl.md`. Fallback: file absent → one `N/A` row after Step 3 and STOP.
|
|
96
99
|
- [ ] **Git History:** Steps 2, 5 — commits since the previous report, fix commits. Fallback: closures and regression-by-commit `Not Checked`.
|
|
97
100
|
- [ ] **Web:** Step 8 — fetch anchor standards. Fallback: versions recorded as unread.
|
|
@@ -101,7 +104,7 @@ Always verify.
|
|
|
101
104
|
Verify only when applicable.
|
|
102
105
|
|
|
103
106
|
- [ ] **Language Server:** Typed language → Step 4 — symbol at current path, caller chain, diagnostics for one file without full typecheck, e.g. Serena, `tsserver`. Fallback: text search. Reach and close needing callers stay hypotheses until corroborated.
|
|
104
|
-
- [ ] **Error Tracker:** Production or preview events exist → Steps 4, 5 — event counts and flows after fix commit, e.g. Sentry, platform logs. Fallback: code only. Event counts `Not Checked`.
|
|
107
|
+
- [ ] **Error Tracker:** Production or preview events exist → Steps 4, 5 — event counts and flows after fix commit, e.g. Sentry, platform logs. Log access: `.agents/context.md` `context-fetch` blocks below `bkit:end` are ready-made read-only readers (`bkit context list-blocks` → each block's `context` and `body`); set a body's `# Customise:` knobs to the audit window; run it via `direnv exec . bash`. Fallback: code only. Event counts `Not Checked`.
|
|
105
108
|
- [ ] **Tracker:** Issues wired → Steps 4, 7 — issue carrying bug, postmortem tracking issue. Fallback: issue pointers `Not Checked`.
|
|
106
109
|
- [ ] **CI Config:** Workflow can fail build → Step 3 — gate on open `CRITICAL` failures. Fallback: gate check `Not Checked`.
|
|
107
110
|
|
|
@@ -114,7 +117,7 @@ Verify only when applicable.
|
|
|
114
117
|
|
|
115
118
|
### Probe Steps
|
|
116
119
|
|
|
117
|
-
- [ ] **Step 1 — Previous Run:**
|
|
120
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
118
121
|
- [ ] **Step 2 — Closures:** Re-verify every FIXED and MIS-FRAMED row from previous report at `HEAD`. Each no longer holding = finding, with what changed. **Done when:** `Closures` holds every row, each failure, and failure rate, or run marked baseline.
|
|
119
122
|
- [ ] **Step 3 — Fold:** Last-row-wins per `key`, not per line. Append-only file (`.gitattributes` `merge=union`); editing a line keeps stale copy beside replacement. Highest numeric `ts` wins (`.bkit/jsonl.md` § Updating a Row). `key` whose `ts` is not a number → last line in file is latest. Open = every winning row with `status != "CLOSED"`, including `TRIAGED` (diagnosed is not fixed), `RESOLVED`, `FIXED`, `SUPERSEDED`, and rows with no `status`. Word that looks finished is not done. `RESOLVED` or `FIXED` row really done → `CLOSED` row in `Expected`, with `references` that closed it. Not skipped because word looks finished. Order: `CRITICAL`, then `HIGH`, then oldest `ts`. Compare `severity` case-insensitively. No `severity` → assign with verdict. Age from row's first `ts`: list every `CRITICAL` older than 7 days and every `HIGH` older than 30 days. Where CI check goes red while any `CRITICAL` row has `status != "CLOSED"`, report whether installed. Green check beside such row = finding (gate not reading ledger). Absent → record in `Checked`. Large open set may be partial pass → record rows verified / rows open. Winning future `ts` that would outrank close → name `key`. No later `ts` invented. **Done when:** `Trend` holds open counts, age-target breaches, gate result, and verified / open, or `N/A` and STOP.
|
|
120
123
|
- [ ] **Step 4 — Source:** No verdict before this step. Under-reporting biases toward closing live bug.
|
|
@@ -237,6 +240,13 @@ PENDING
|
|
|
237
240
|
|
|
238
241
|
PENDING
|
|
239
242
|
|
|
243
|
+
### Suggested Issues
|
|
244
|
+
|
|
245
|
+
<!-- prettier-ignore -->
|
|
246
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
247
|
+
| --- | --- | --- | --- | --- | --- |
|
|
248
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
249
|
+
|
|
240
250
|
### Process Improvements
|
|
241
251
|
|
|
242
252
|
PENDING
|
|
@@ -250,6 +260,7 @@ PENDING
|
|
|
250
260
|
- [ ] **Series:** open count and the recurrence ranking beside previous report's numbers, or marked baseline.
|
|
251
261
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
252
262
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
263
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
253
264
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
254
265
|
|
|
255
266
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -79,11 +79,14 @@ One per flake. Root-cause class (`Mechanism`) = evidence; these = outcomes. Quar
|
|
|
79
79
|
|
|
80
80
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
81
81
|
|
|
82
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
83
|
+
|
|
82
84
|
#### Core
|
|
83
85
|
|
|
84
86
|
Always verify.
|
|
85
87
|
|
|
86
|
-
- [ ] **
|
|
88
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
89
|
+
- [ ] **CI History:** Steps 2, 3 — workflow runs since previous report, e.g. `gh run list`, Actions API, GitLab pipelines. Log access: `.agents/context.md` `context-fetch` blocks below `bkit:end` are ready-made read-only readers (`bkit context list-blocks` → each block's `context` and `body`); set a body's `# Customise:` knobs to the audit window; run it via `direnv exec . bash`. Fallback: local repeats only; ranked CI list `Not Checked`.
|
|
87
90
|
- [ ] **Repeat Runner:** Steps 4, 7, 8 — repeat with retries off, e.g. Playwright `--repeat-each`, pytest-repeat. Fallback: rate `Not Checked`; do not mark `FIX NAMED` on unmeasured flake unless mechanism deterministic and report says rate unknown.
|
|
88
91
|
- [ ] **Web:** Steps 6, 13 — vendor status pages and anchor standards. Fallback: incidents and versions recorded as unread.
|
|
89
92
|
|
|
@@ -105,7 +108,7 @@ Verify only when applicable.
|
|
|
105
108
|
|
|
106
109
|
### Probe Steps
|
|
107
110
|
|
|
108
|
-
- [ ] **Step 1 — Previous Run:**
|
|
111
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
109
112
|
- [ ] **Step 2 — List:** Before triage, list every test with runner status flaky (failed, passed on retry) and every pass-only-on-retry, across every workflow since previous report. Also list fail one run/surface, pass another: with retries off, `flaky` never appears; a noticed red run is input, not the list. Rank by flake rate (flaky or failed runs ÷ runs) and reruns caused. Per open flake: first-seen, last-seen, age. Report median age of open flakes. Red run absent from list → detection finding. Triage in rank order. **Done when:** ranked list, ages, median in `Flakes`.
|
|
110
113
|
- [ ] **Step 3 — Pin:** Per triaged flake: run id, commit, job that failed vs downstream-only. Control run and valid scope. Prior red run = control only when failure mode differs — auth failure redirecting every lane to sign-in is control for tenant-minting lane only. **Done when:** run, job, control in `Pin`.
|
|
111
114
|
- [ ] **Step 4 — Rate:** Spec with repeat-each, retries off, at CI worker count: k failures in N runs. Repeat with CPU throttled/loaded. Record N, k, workers, commit. Without measured rate, nothing to compare fix against (Step 11). **Done when:** N, k, workers, commit in `Rate`, or why not measured.
|
|
@@ -116,7 +119,7 @@ Verify only when applicable.
|
|
|
116
119
|
- [ ] **Step 9 — Native:** Device-runner flake (e.g. Maestro, EAS) → Steps 5–6 target browser; also boot finished before flow, device-cloud retries, animation/keyboard timing. Record artifact locations (screenshots, recordings, logs). **Done when:** answers in `Native`, or skip with evidence.
|
|
117
120
|
- [ ] **Step 10 — Mechanism:** Teardown deletes rows event points at → most diagnoses corroborated by mechanism, not row. Distinguish "fits" vs "proven". One root-cause class: async wait, concurrency, order dependency, shared fixture state, resource leak, network, external service, time and date, randomness, unordered collection, infrastructure, or other (describe). Luo et al.: async wait, concurrency, order dependency ≈ 78% classified. Per-class counts across open flakes = series. Before timing mechanism → floating-promise/missing-await rules on spec/helpers (e.g. `@typescript-eslint/no-floating-promises`, `playwright/missing-playwright-await`); record hits so missing `await` is not hidden behind timing. Real defect behind flake → verdict `BUG`. **Done when:** class, what is not established, await hits in `Mechanism`.
|
|
118
121
|
- [ ] **Step 11 — Fix:** Name cause and change that removes it; making change = next step. Never `.skip`, quarantine, delete assertion, widen timeout. Before trusting the repaired test, the next step puts the bug back, watches the test go red, restores, and confirms `git diff` on the source file is empty. That is not optional on anything touching row-level access control or a cross-tenant boundary. Shared helper/fixture change = signature change: references first. One-spec fix via shared fixture → next unattributable failure. Put bug back proves deterministic fix. Flaky fix also repeats Step 4: at rate p, ~3/p runs with no failure → 95% confidence rate below p (rule of three); report states count. One verdict from `Verdicts`. **Done when:** verdict, change, or why none, run count in `Fix`.
|
|
119
|
-
- [ ] **Step 12 — Route:** No finding without re-checkable evidence: `file:line`, or query/run id/measurement + tool. Language-server empty inconclusive → text search corroboration. Report names destinations (sending = next step). Every flake → `docs/failures.jsonl` row (class, rate, age). Named mechanism → recommended fix + proof runs (Step 11). Real defect → own issue + ledger row. Fix waits on decision → named decision. Mechanism error tracker also saw → `bkit-audit-logs`. Retry-pass/warning CI annotated → `bkit-audit-annotations`. Missing trace/capture, third-party stub, missing `await` → recommended change (Steps 5, 10). **Done when:** every finding's Expected names destination, including ledger row.
|
|
122
|
+
- [ ] **Step 12 — Route:** No finding without re-checkable evidence: `file:line`, or query/run id/measurement + tool. Language-server empty inconclusive → text search corroboration. Report names destinations (sending = next step); the audit agent writes `docs/failures.jsonl` rows. Every flake → `docs/failures.jsonl` row (class, rate, age). Named mechanism → recommended fix + proof runs (Step 11). Real defect → own issue + ledger row. Fix waits on decision → named decision. Mechanism error tracker also saw → `bkit-audit-logs`. Retry-pass/warning CI annotated → `bkit-audit-annotations`. Missing trace/capture, third-party stub, missing `await` → recommended change (Steps 5, 10). **Done when:** every finding's Expected names destination, including ledger row.
|
|
120
123
|
- [ ] **Step 13 — Standards:** Confirm current version per anchor standard; read changes since previous report's version. `Standards Coverage`: row per Luo et al. class + per flake feature the runner offers (e.g. retries, flaky status, repeat-each, fail-on-flaky, sequence shuffle) → checked (name step) / not applicable (why) / finding. Unapplied recommended practice → adopt (propose step) / reject (why) / not applicable. Do not change steps during run. **Done when:** no empty row.
|
|
121
124
|
- [ ] **Step 14 — Anything Else:** (a) Since previous report's commit, everything touching test stability (browser/unit config: retries, workers, timeouts, trace; shared fixtures/helpers; CI workflows/runners; preview/DB for E2E; device flows) → one-line verdict each. (b) Bounded ~30 min: senior test-infra flags no step asks; read since previous run (incidents, `bkit-audit-logs`, new `docs/failures.jsonl`, vendor status) — what would no step catch? **Done when:** `Beyond the Steps` holds both passes, or `none`. Class seen on 2 runs → propose step in `Process Improvements`.
|
|
122
125
|
|
|
@@ -215,6 +218,13 @@ PENDING
|
|
|
215
218
|
|
|
216
219
|
PENDING
|
|
217
220
|
|
|
221
|
+
### Suggested Issues
|
|
222
|
+
|
|
223
|
+
<!-- prettier-ignore -->
|
|
224
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
225
|
+
| --- | --- | --- | --- | --- | --- |
|
|
226
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
227
|
+
|
|
218
228
|
### Process Improvements
|
|
219
229
|
|
|
220
230
|
PENDING
|
|
@@ -228,6 +238,7 @@ PENDING
|
|
|
228
238
|
- [ ] **Series:** each open flake's rate and the median age beside previous report's numbers, or marked baseline.
|
|
229
239
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
230
240
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
241
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
231
242
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
232
243
|
|
|
233
244
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -71,10 +71,13 @@ One per candidate.
|
|
|
71
71
|
|
|
72
72
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
73
73
|
|
|
74
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
75
|
+
|
|
74
76
|
#### Core
|
|
75
77
|
|
|
76
78
|
Always verify.
|
|
77
79
|
|
|
80
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
78
81
|
- [ ] **Sibling Reports:** Steps 2, 3 — newest report per other `bkit-audit-*` slug under `docs/audits/`, else that template's `Process Improvements`. Fallback: templates only; a missing report = that audit `Not Checked`.
|
|
79
82
|
- [ ] **Ledgers:** Steps 3, 10, 11 — `docs/failures.jsonl`, `docs/decisions.jsonl`, `docs/deferrals.jsonl`. Fallback: incident evidence and prior rejections `Not Checked`.
|
|
80
83
|
- [ ] **Repo Gates:** Steps 7, 9, 12, 13, 15 — workflows, hook directory, gate scripts, allowlist files, e.g. `.github/workflows`, `.husky`, `.gitlab-ci.yml`, every `bkit:gate:*` script in `package.json`. Fallback: none; those items `Not Checked`.
|
|
@@ -97,7 +100,7 @@ Verify only when applicable.
|
|
|
97
100
|
|
|
98
101
|
### Probe Steps
|
|
99
102
|
|
|
100
|
-
- [ ] **Step 1 — Previous Run:**
|
|
103
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
101
104
|
- [ ] **Step 2 — Inventory:** Collect gate candidates from each other `bkit-audit-*` report's `Process Improvements` (else that template's), without re-deriving. Record source slug. Two audits proposing one guard = one candidate, both sources — not two work items. Re-grep of a swept class + owner-allowlist over same literals = one guard. **Done when:** `Inventory` row per candidate: source slug · one-line guard.
|
|
102
105
|
- [ ] **Step 3 — Widen:** Sibling sections are one source. Also add, record source each: (a) every defect class in `docs/failures.jsonl` on ≥2 rows, whether or not a sibling proposed a gate; (b) standard gates nobody proposed, each vs tree — secret-scanning push protection, code scanning (e.g. CodeQL), dependency-review check, workflow linters (e.g. `actionlint`, `zizmor`), lockfile lint, test-runner forbid-only flag (e.g. Playwright `--forbid-only`), typechecking of test files, OpenSSF Scorecard checks; (c) policy-as-code for config with no type system (e.g. JSON Schema, Conftest) — bot config, job-runner triggers and concurrency, hosting config, mobile build config, workflow YAML. Class (c) is what `bkit-audit-testing` flags. Already enforced → `Health`, not a second candidate. **Done when:** every added row in `Inventory` with source.
|
|
103
106
|
- [ ] **Step 4 — Score:** Per candidate, six axes. Fails if fail condition holds.
|
|
@@ -228,6 +231,13 @@ PENDING
|
|
|
228
231
|
|
|
229
232
|
PENDING
|
|
230
233
|
|
|
234
|
+
### Suggested Issues
|
|
235
|
+
|
|
236
|
+
<!-- prettier-ignore -->
|
|
237
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
238
|
+
| --- | --- | --- | --- | --- | --- |
|
|
239
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
240
|
+
|
|
231
241
|
### Process Improvements
|
|
232
242
|
|
|
233
243
|
PENDING
|
|
@@ -241,6 +251,7 @@ PENDING
|
|
|
241
251
|
- [ ] **Series:** verdict counts, `Health`, and allowlist sizes beside previous report's numbers, or marked baseline.
|
|
242
252
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
243
253
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
254
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
244
255
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
245
256
|
|
|
246
257
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -113,10 +113,13 @@ Finding `CRITICAL`–`LOW` in another audit does not set priority. Two rules do.
|
|
|
113
113
|
|
|
114
114
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
115
115
|
|
|
116
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
117
|
+
|
|
116
118
|
#### Core
|
|
117
119
|
|
|
118
120
|
Always verify.
|
|
119
121
|
|
|
122
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
120
123
|
- [ ] **Tracker Read:** Steps 2, 5, 10 — every board item with lane, size, and priority, e.g. `gh project item-list`, Linear issues. Fallback: none; unread items `Not Checked`, proposal does not cover them.
|
|
121
124
|
- [ ] **Open-Issue List:** Steps 2, 10 — every open issue, every repo / team board draws from, read separately from board, e.g. `gh issue list --state open`, Linear team issues. Fallback: none; membership `Not Checked`, proposal covers board items only.
|
|
122
125
|
- [ ] **Issue Bodies:** Steps 3, 4, 6, 8 — title, labels, body, parent, milestone, blocked-by, e.g. `gh issue view`, Linear issue API. Fallback: titles only; each such rank low-confidence in `Judgement Calls`, links from structured fields only.
|
|
@@ -138,7 +141,7 @@ Verify only when applicable.
|
|
|
138
141
|
|
|
139
142
|
### Probe Steps
|
|
140
143
|
|
|
141
|
-
- [ ] **Step 1 — Previous Run:**
|
|
144
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
142
145
|
- [ ] **Step 2 — Board:** Read every item. Assert read not truncated (count below limit; at limit → raise and re-read). Record lane names, size and priority field ids and option ids from live field list. Unused numeric estimate stays unused; size = estimate. Membership: diff open-issue list vs board. Open issue not on board = off-board; closed issue outside done lane = stale card; list each by number. Off-board stays in scope: Steps 3–10 treat as item with no lane, size, priority. **Done when:** item count, lanes, field ids, off-board, stale cards in `Board`, or `N/A` and STOP.
|
|
143
146
|
- [ ] **Step 3 — Triage:** Read every open issue, not titles only. Short body prefix OK when summary front-loaded; open umbrella/milestone issues in full. While reading: stale = no activity 90 days and body no longer true of code/plan (standing issue never stale for age); duplicate = same change twice or one wholly inside another; `Low` on 3+ passes and still true = won't-do candidate (standing excluded); ready-lane item missing acceptance criteria, defect reproduction, or test to write = not ready (still gets size/priority). Note each Rule 1 override for Step 6 with label correction. List stale/duplicate with merge target or stale reason. Closing = owner; this run does not close. Monthly clock → stale only. **Done when:** `Triage` lists stale, duplicates, won't-do, not-ready, overrides to rank.
|
|
144
147
|
- [ ] **Step 4 — Links:** Per open issue record parent, milestone, blocked-by. Blocked-by: read all three — structured links (`Link API`), body block build gates on (e.g. GitHub `###### bkit:blocked-by start` / `end`), body prose ("blocked by #N", "after #N"). Flag: (1) Parent — child listed in parent body, no link (or reverse); open child, closed parent; open non-standing parent, all children closed → `Judgement Calls`. (2) Milestone — closed / past-due milestone with open issues; child with no or later milestone than parent; body / title names milestone, field empty; open milestone, no open issues. (3) Blocked-by — prose only; structured link vs body block disagree; closed blocker still listed; blocked issue in ready or later lane, blocker open; blocker off-board; cycle. Edges → Step 5 blocker rule, Step 6 parent / child moves. **Done when:** `Links` holds per-kind counts (with / without parent, milestone, blocker) and every flag, or `none`.
|
|
@@ -251,6 +254,13 @@ PENDING
|
|
|
251
254
|
|
|
252
255
|
PENDING
|
|
253
256
|
|
|
257
|
+
### Suggested Issues
|
|
258
|
+
|
|
259
|
+
<!-- prettier-ignore -->
|
|
260
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
261
|
+
| --- | --- | --- | --- | --- | --- |
|
|
262
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
263
|
+
|
|
254
264
|
### Process Improvements
|
|
255
265
|
|
|
256
266
|
PENDING
|
|
@@ -264,6 +274,7 @@ PENDING
|
|
|
264
274
|
- [ ] **Series:** every series beside previous report's numbers, or marked baseline.
|
|
265
275
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
266
276
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
277
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
267
278
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
268
279
|
|
|
269
280
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -77,10 +77,13 @@ Taxonomy names match `bkit-audit-red-team`, so a finding here and its live proof
|
|
|
77
77
|
|
|
78
78
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
79
79
|
|
|
80
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
81
|
+
|
|
80
82
|
#### Core
|
|
81
83
|
|
|
82
84
|
Always verify.
|
|
83
85
|
|
|
86
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
84
87
|
- [ ] **Code Search:** Steps 2–12, 14 — text search and language server for callers (e.g. ripgrep, Serena, `tsserver`). Prompts often markdown; tool names often strings. Language-server hits under-report; empty inconclusive → corroborate with text search and name the tool. Fallback: text search only; name it per claim.
|
|
85
88
|
- [ ] **Taxonomy Skill:** Steps 2–9, 13 — skill covering LLM and agentic top 10 editions in Anchor Standards (e.g. `owasp-security`). Categories it lacks → `Checked`. Fallback: read anchor standards directly.
|
|
86
89
|
- [ ] **Web:** Steps 4, 9, 13 — tool-server spec, provider terms, anchor standards. Fallback: regime and version changes recorded as unread.
|
|
@@ -101,7 +104,7 @@ Verify only when applicable.
|
|
|
101
104
|
|
|
102
105
|
### Probe Steps
|
|
103
106
|
|
|
104
|
-
- [ ] **Step 1 — Previous Run:**
|
|
107
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
105
108
|
- [ ] **Step 2 — Inventory:** Start from code, not remembered caller list. Find every model-package import outside that package and outside tests; list prompt-set directories. Load taxonomy skill; categories it lacks vs anchor editions → `Checked`. Per model call and tool, one `Inventory` row: input and who controls it · model-group name at call site (choice, deprecation, catalogue, zero-retention routing → `bkit-audit-models`; do not score here) · tools offered · what a tool call writes, and to which tenant · DB client the write runs as · tenant data in prompt · output reaching someone outside tenant. Diff table vs previous run. New call site, prompt set, or tool read first. Per surface, AI impact-assessment record with named owner (ISO/IEC 42001 6.1.4, ISO/IEC 42005, NIST AI RMF MAP). Update from this table; do not rebuild. No record = finding. Checked-in inventory failing when new call site has no row → propose under `Process Improvements`; do not add this run. **Done when:** `Inventory` lists every call and tool, or `N/A` and STOP.
|
|
106
109
|
- [ ] **Step 3 — Capability:** Per tool: does caller need it, what bounds it? Model writes from untrusted inbound → which rows: only from same message/thread, or any tenant row? State outcome of instruction to delete everything matching a phrase. "Any row" = finding, ranked by one-message removal scope. Outside-tool credential tenant-wide → read-only limit possible? If not, design question in `Governance`, not defect, unless code contradicts `docs/decisions.jsonl`. Read-only tools write nothing; destructive tools marked destructive. Host skipping confirmation on read-only tools → wrong annotation removes human check. Allowlist subtracting fields still subtracts; contract test pins; new tools through same filter. Tenant id from credential row, never tool arguments. Check every tool since previous run. Similarity search scoped by tenant in query (LLM08). Every embedding query filters tenant in query itself (privileged role bypasses RLS). Embedding regenerated on source edit, deleted on row delete. **Done when:** `Capability` states inbound bound and verdict for annotations, allowlist, tenant id, embedding scope.
|
|
107
110
|
- [ ] **Step 4 — Governance:** Outside model holds tool-server credential → check against protocol authorization and security best practices. Record auth model. Where spec says HTTP server with authorization should use OAuth with protected-resource metadata and audience-bound tokens, static bearer = design question in `Governance`, not defect, unless code contradicts `docs/decisions.jsonl`. Token lifecycle: default lifetime, scopes, rotation/revocation, per-tool-call logging per token. Server never passes received token downstream. Snapshot every tool name, description, annotations; diff previous run. Changed tool description = supply-chain risk (ASI04): outside model reads as instructions → needs reviewer. No tool server → skip with evidence. **Done when:** `Governance` holds auth, lifecycle, passthrough, description diff, or skip.
|
|
@@ -215,6 +218,13 @@ PENDING
|
|
|
215
218
|
|
|
216
219
|
PENDING
|
|
217
220
|
|
|
221
|
+
### Suggested Issues
|
|
222
|
+
|
|
223
|
+
<!-- prettier-ignore -->
|
|
224
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
225
|
+
| --- | --- | --- | --- | --- | --- |
|
|
226
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
227
|
+
|
|
218
228
|
### Process Improvements
|
|
219
229
|
|
|
220
230
|
PENDING
|
|
@@ -230,6 +240,7 @@ PENDING
|
|
|
230
240
|
- [ ] **Registry:** model choice, deprecation, and catalogue checks point at `bkit-audit-models` and are not scored here.
|
|
231
241
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
232
242
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
243
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
233
244
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
234
245
|
|
|
235
246
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|