@cbartens/bkit 5.2.2 → 5.2.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.bkit/README.md +4 -2
- package/.bkit/core/.agents/context.md +2 -2
- package/.bkit/core/.agents/plugins/bkit/hooks/bkit-guardrails.cjs +7 -1
- package/.bkit/core/.bkit/README.md +3 -1
- package/.bkit/core/.bkit/comparison.md +114 -45
- package/.bkit/core/.bkit/install.md +6 -6
- package/.bkit/core/.bkit/jsonl.md +3 -3
- package/.bkit/core/.bkit/prompts.md +6 -5
- package/.bkit/core/.bkit/providers.md +5 -4
- package/.bkit/core/.bkit/templates.md +30 -11
- package/.bkit/core/.bkit/workflows.md +14 -7
- package/.bkit/core/.codex/hooks/bkit-guardrails.cjs +5 -1
- package/.bkit/core/.cursor/hooks/bkit-guardrails.cjs +8 -2
- package/.bkit/core/docs/templates/bkit-audit-accessibility.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-adoption.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-agents.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-aio-seo.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-analytics.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-annotations.md +15 -4
- package/.bkit/core/docs/templates/bkit-audit-app-performance.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-app-store.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-architecture.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-ci-performance.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-config.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-cost.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-coverage.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-database.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-dead-code.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-deferrals.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-dependencies.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-documentation.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-duplication.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-failures.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-flakes.md +15 -4
- package/.bkit/core/docs/templates/bkit-audit-gates.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-issues.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-llm.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-logs.md +25 -13
- package/.bkit/core/docs/templates/bkit-audit-models.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-privacy.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-prompts.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-red-team.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-reliability.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-repo.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-security.md +15 -4
- package/.bkit/core/docs/templates/bkit-audit-slop.md +13 -2
- package/.bkit/core/docs/templates/bkit-audit-testing.md +14 -3
- package/.bkit/core/docs/templates/bkit-audit-usability.md +13 -2
- package/.bkit/core/docs/templates/bkit-plan.md +1 -1
- package/.bkit/core/engine/checks/tips/catalogue.ts +10 -0
- package/.bkit/core/engine/commands/install/check.ts +3 -11
- package/.bkit/core/engine/commands/install/doctor.ts +2 -3
- package/.bkit/core/engine/commands/install/uninstall.ts +60 -44
- package/.bkit/core/engine/commands/install/upgrade.ts +13 -11
- package/.bkit/core/engine/commands/workflow/branches.ts +72 -9
- package/.bkit/core/engine/commands/workflow/repo.ts +16 -2
- package/.bkit/core/engine/commands/workflow/workflows.ts +4 -3
- package/.bkit/core/engine/install/hosts.ts +14 -4
- package/.bkit/core/engine/ledgers/telemetry.ts +3 -4
- package/.bkit/core/engine/shared/bkit-latest.ts +164 -0
- package/.bkit/core/engine/shared/timeouts.ts +6 -0
- package/.bkit/core/hooks/guardrails-read.js +4 -9
- package/.bkit/core/hooks/guardrails-tiers.js +312 -28
- package/.bkit/core/prompts/bkit-audit-generate.md +21 -5
- package/.bkit/core/workflows/bkit-build.md +1 -1
- package/.bkit/core/workflows/bkit-debug.md +1 -1
- package/.bkit/core/workflows/bkit-design.md +1 -1
- package/.bkit/core/workflows/bkit-plan.md +1 -1
- package/.bkit/core/workflows/bkit-ship.md +1 -1
- package/.bkit/shared/.claude/settings.json +19 -0
- package/.bkit/shared/.opencode/opencode.json +21 -1
- package/.bkit/shared/docs/deployments.md +14 -0
- package/.bkit/templates/.github/workflows/bkit-ci.yml +9 -0
- package/.bkit/templates/.husky/pre-push +7 -0
- package/.bkit/templates/context/bkit-cloudrun.md +28 -19
- package/.bkit/templates/context/bkit-expo.md +30 -13
- package/.bkit/templates/context/bkit-inngest.md +54 -17
- package/.bkit/templates/context/bkit-resend.md +36 -8
- package/.bkit/templates/context/bkit-sentry.md +11 -4
- package/.bkit/templates/context/bkit-stripe.md +39 -22
- package/.bkit/templates/context/bkit-supabase.md +45 -20
- package/.bkit/templates/context/bkit-vercel.md +45 -16
- package/.bkit/templates/scripts/ci/accepted-advisories.json +7 -0
- package/.bkit/templates/scripts/ci/check-dependency-audit.mjs +200 -0
- package/.bkit/templates/scripts/ci/check-dependency-audit.test.mjs +127 -0
- package/.bkit/templates/scripts/cleanup/bkit-tmp-prune.mjs +115 -0
- package/CHANGELOG.md +56 -0
- package/README.md +76 -20
- package/dist/checks/tips/catalogue.js +9 -0
- package/dist/commands/install/check.js +3 -15
- package/dist/commands/install/doctor.js +2 -1
- package/dist/commands/install/uninstall.js +52 -45
- package/dist/commands/install/upgrade.js +13 -11
- package/dist/commands/workflow/branches.js +64 -10
- package/dist/commands/workflow/repo.js +14 -2
- package/dist/commands/workflow/workflows.js +4 -3
- package/dist/install/hosts.js +11 -4
- package/dist/shared/bkit-latest.js +150 -0
- package/dist/shared/timeouts.js +6 -0
- package/package.json +1 -1
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -70,7 +70,7 @@ One per unresolved issue. Verdict decides everything downstream.
|
|
|
70
70
|
|
|
71
71
|
- **GENUINE ERROR:** unhandled failure, or handled with wrong handling. Root-cause it. Fix = next step.
|
|
72
72
|
- **WRONG LEVEL:** working as intended, filed at error: 2xx response, guard did its job, or docblock documents branch. Recommend level change. Do not edit call site.
|
|
73
|
-
- **NOISE:** vendor or infrastructure, no app call site (SDK internals, tunnel egress, platform insight). Recommend tracker-side filter (e.g. `beforeSend` denylist) and
|
|
73
|
+
- **NOISE:** vendor or infrastructure, no app call site (SDK internals, tunnel egress, platform insight). Recommend tracker-side filter (e.g. `beforeSend` denylist) and `docs/noise.md` row. Do not add filter here. Denylist = vendor frames with no call site, not app code.
|
|
74
74
|
- **OTHER:** fits none. Say why and what the issue needs. Same `OTHER` on 2 runs → propose new verdict in `Process Improvements`.
|
|
75
75
|
|
|
76
76
|
A test-origin tag narrows the lane. It does not make something noise, and it does not lower severity.
|
|
@@ -79,10 +79,13 @@ A test-origin tag narrows the lane. It does not make something noise, and it doe
|
|
|
79
79
|
|
|
80
80
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
81
81
|
|
|
82
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
83
|
+
|
|
82
84
|
#### Core
|
|
83
85
|
|
|
84
86
|
Always verify.
|
|
85
87
|
|
|
88
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
86
89
|
- [ ] **Error Tracker:** Steps 3–9 — read-only issue, event, and alert APIs, e.g. tracker MCP, REST. Fallback: skip tracker steps; runtime-log steps still run; tracker sections `Not Checked`. Never "clean".
|
|
87
90
|
- [ ] **Web:** Step 13 — fetch anchor standards. Fallback: versions recorded as unread.
|
|
88
91
|
|
|
@@ -90,7 +93,7 @@ Always verify.
|
|
|
90
93
|
|
|
91
94
|
Verify only when applicable.
|
|
92
95
|
|
|
93
|
-
- [ ] **Runtime Logs:** Hosted platforms → Step 11 — read-only log APIs, e.g.
|
|
96
|
+
- [ ] **Runtime Logs:** Hosted platforms → Step 11 — read-only log APIs, e.g. vendor CLIs, MCPs. Log access: `.agents/context.md` `context-fetch` blocks below `bkit:end` are ready-made read-only readers (`bkit context list-blocks` → each block's `context` and `body`); set a body's `# Customise:` knobs to the audit window; run it via `direnv exec . bash`. Skipped credential = not read, never clean. Fallback: each skipped source named in `Not Checked`.
|
|
94
97
|
- [ ] **Language Server:** Typed language → Step 7 — references, e.g. Serena, `tsserver`. Fallback: text search only; name it on every blast-radius row.
|
|
95
98
|
- [ ] **Symbol Maps:** Release builds → Step 4 — source maps and native symbols. Fallback: unsymbolicated = finding; call-site table marked unbuildable.
|
|
96
99
|
- [ ] **User Reports:** Feedback channel exists → Step 10 — tracker user feedback, support inbox, store reviews. Fallback: `Not Checked`.
|
|
@@ -104,15 +107,16 @@ Verify only when applicable.
|
|
|
104
107
|
|
|
105
108
|
### Probe Steps
|
|
106
109
|
|
|
107
|
-
- [ ] **Step 1 — Previous Run:**
|
|
110
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
108
111
|
- [ ] **Step 2 — Detect:** Find error tracker, runtime logs, or both. **Done when:** evidence in `Checked`, or `N/A` and STOP.
|
|
109
112
|
- [ ] **Step 3 — Board:** Same figures every run for comparison. Window = since previous report; also trailing-30-day when that is tracker default:
|
|
110
113
|
- Unresolved count, and issue short ids.
|
|
111
114
|
- Regressed count.
|
|
112
115
|
- Events by environment. Name each. Call out production.
|
|
113
116
|
- First seen, per issue, UTC.
|
|
117
|
+
- Reported vs platform failures. Per platform re-reporting failures to the tracker (e.g. job runner, host): platform failure count beside tracker event count, same window. Ratio < 1 = reports lost.
|
|
114
118
|
|
|
115
|
-
Production count > zero → reprioritize everything. No real user traffic → numbers describe CI/test traffic, not user impact; report says so. Resolve release SHA per issue (`git rev-parse`, `git branch --contains`) before branch attribution. Integration-branch HEAD event = dev-side bug, not PR-under-review defect. Release health per client (e.g. web, native): crash-free sessions and crash-free users, per release and environment; latest-release adoption. Low traffic → describes CI; report says so. Previous report expected fix → confirm held and duration. Miss in window = finding. "Shipped" fix absent from default branch = still live = finding. **Done when:** `Board` holds the
|
|
119
|
+
Production count > zero → reprioritize everything. No real user traffic → numbers describe CI/test traffic, not user impact; report says so. Resolve release SHA per issue (`git rev-parse`, `git branch --contains`) before branch attribution. Integration-branch HEAD event = dev-side bug, not PR-under-review defect. Release health per client (e.g. web, native): crash-free sessions and crash-free users, per release and environment; latest-release adoption. Low traffic → describes CI; report says so. Previous report expected fix → confirm held and duration. Miss in window = finding. "Shipped" fix absent from default branch = still live = finding. **Done when:** `Board` holds the five figures, release health, and those confirmations.
|
|
116
120
|
|
|
117
121
|
- [ ] **Step 4 — Instrument:** Read tracker project settings, read-only, against anchor standards.
|
|
118
122
|
- Symbolication. One recent production event and one non-production: frames show original file/line, event carries debug ids, release set. Native client → Hermes maps, dSYM, ProGuard uploads where that client exists. Unsymbolicated stack → call-site table unbuildable = finding.
|
|
@@ -132,31 +136,31 @@ Verify only when applicable.
|
|
|
132
136
|
|
|
133
137
|
- [ ] **Step 6 — Classify:** One verdict per unresolved issue. Tracker performance issues (e.g. N+1, slow DB spans) and Web Vitals regressions: not classified. Record count on `Board`; name defect-like ones for `bkit-audit-app-performance`. Flake mechanism → note for `bkit-audit-flakes`. **Done when:** every unresolved issue has verdict in `Issues`; performance count on `Board`.
|
|
134
138
|
- [ ] **Step 7 — Root Cause:** Each `GENUINE ERROR` and any issue whose verdict depends on call site. Message = symptom. Read call site → path → blast radius. Call-site table = artefact: callers sharing broken mechanism vs not. Language server, where available: stack symbol → every referencing symbol → diagnostics for that file. Pace reference queries; partial answer = indistinguishable from real. Re-run surprising low count. Empty = inconclusive → corroborate with text search; say which count from where. Ask why no test caught it. Every suite mocking SDK = second finding. Verify constraints against live system, not memory: actual predicate, rolled-back insert, host vs published IP ranges. Say what is unverifiable. Teardown often deletes rows an event points at → non-production findings corroborated by mechanism, not row. A mechanism that fits is not a confirmed cause. **Done when:** each `GENUINE ERROR` row in `Issues` has root cause, call-site table, the method that produced it, what is not established, and the file a later change would touch.
|
|
135
|
-
- [ ] **Step 8 — Level:** Each `WRONG LEVEL`: recommend error-level log → warning, plus
|
|
136
|
-
- [ ] **Step 9 — Disposition:** Recommend resolve, archive, or leave open and why, per issue. This run does not resolve or archive. Recommend resolve only when code that files the issue changed and shipped. Resolving ahead of fix hides only signal on rails with no other coverage. **Done when:** `Issues` states resolve, archive, or leave open for every issue.
|
|
139
|
+
- [ ] **Step 8 — Level:** Each `WRONG LEVEL`: recommend error-level log → warning, plus `docs/noise.md` row (call site, reason). Do not edit. Do not recommend tracker denylist for app code. Before downgrading, check hypothesis that would make it real defect; record check. Read both symbols; do not search column name. Fingerprint explosion: message with varying payload (ids, arrays) needs stable grouping key and pinned test before left in stream. **Done when:** each `WRONG LEVEL` row names the level change, the hypothesis checked, any fingerprint gap, and the file a later change would touch.
|
|
140
|
+
- [ ] **Step 9 — Disposition:** Recommend resolve, archive, or leave open and why, per issue. This run does not resolve or archive. Recommend resolve only when code that files the issue changed and shipped. Resolving ahead of fix hides only signal on rails with no other coverage. Archived / muted issues too: per `docs/noise.md` archive or mute row, compare window events with the row's recorded trigger. Past trigger → recommend un-archive. **Done when:** `Issues` states resolve, archive, or leave open for every issue; every archive / mute row checked against its trigger.
|
|
137
141
|
- [ ] **Step 10 — Invisible:** Tracker only sees errors that reach it. Caught then neither logged nor rethrown = invisible to caller and this sweep. Quiet board ≠ evidence shape absent. Look only in code behind this sweep's issues (`bkit-audit-reliability` sweeps rest):
|
|
138
142
|
- `catch` blocks that neither rethrow, return error, nor log at warn or above. Start with DB writes and outbound calls.
|
|
139
143
|
- Client calls whose error is never read: result discarded, or success destructured without error.
|
|
140
144
|
- Promises started without `await` or rejection handler on write path.
|
|
141
145
|
|
|
142
|
-
Text search finds candidates; read each. Deliberate best-effort (telemetry, fire-and-forget notification) OK when code says so. Real one →
|
|
146
|
+
Text search finds candidates; read each. Deliberate best-effort (telemetry, fire-and-forget notification) OK when code says so. Real one → `docs/failures.jsonl` row, which the audit agent writes; Expected names `bkit-audit-database` when miss is grant, policy, or write role. Users see same shape: tracker user feedback, support inbox, store reviews since previous report. User-reported failure with no matching event = this class. Next step = runtime half: swallowed DB error may appear in DB log. **Done when:** `Invisible` lists each candidate read, and its verdict.
|
|
143
147
|
|
|
144
|
-
- [ ] **Step 11 — Runtime Logs:** Read each runtime-log source in `Project Instructions` for whole window since previous report, not tool default short window. Record retention from plan page and window part source could not cover. Retention < cadence = finding → drain or shorter cadence. Skipped reader (credential, tool, block) = not read, never clean. No reader → vendor console by hand, or not read and why. Note whether server logs share one structured shape (request id, trace id, hashed tenant id, error code) so queries are filters not text search; tracker log product or OTel log exporter in use. Per log error:
|
|
148
|
+
- [ ] **Step 11 — Runtime Logs:** Read each runtime-log source in `Project Instructions` and each `context=runtime` block in `.agents/context.md` for whole window since previous report, not tool default short window. Record retention from plan page and window part source could not cover. Retention < cadence = finding → drain or shorter cadence. Skipped reader (credential, tool, block) = not read, never clean. No reader → vendor console by hand, or not read and why. Note whether server logs share one structured shape (request id, trace id, hashed tenant id, error code) so queries are filters not text search; tracker log product or OTel log exporter in use. Per log error:
|
|
145
149
|
- Match tracker first (timestamps; trace id joins). Covered → note platform evidence on issue. Same incident, not second finding.
|
|
146
150
|
- Group by time across sources. Cluster within few minutes = one outage; write once.
|
|
147
151
|
- Tracker empty → gap = finding. Noise (why), or failure that should have been reported. Second → Expected names signal: log at call site, upstream-outage classifier pattern, or health-endpoint check. Monitor/capacity alert = next step, not this run.
|
|
148
152
|
- Never copy log line free text. PII-safe fields reader already selects.
|
|
149
153
|
|
|
150
|
-
Check correlation without clock. Every inbound-request → background-work path: one trace (e.g. W3C `traceparent`, tracker trace header) spans host, job runner, DB calls, in tracker and job-runner traces. Broken hop = finding. Without it, time grouping = only join. Platform failure re-reported to tracker → counts must match; platform failure with no tracker event = broken reporter. **Done when:** `Runtime Logs` row per source: read or not (why) · window covered · retention · error count · matched tracker issues. Correlation recorded.
|
|
154
|
+
Check correlation without clock. Every inbound-request → background-work path: one trace (e.g. W3C `traceparent`, tracker trace header) spans host, job runner, DB calls, in tracker and job-runner traces. Broken hop = finding. Without it, time grouping = only join. Platform failure re-reported to tracker → counts must match; platform failure with no tracker event = broken reporter. **Done when:** `Runtime Logs` row per source: read or not (why) · window covered · retention · error count · matched tracker issues. Correlation recorded. `Board` holds reported-vs-platform ratio per reporting platform.
|
|
151
155
|
|
|
152
156
|
- [ ] **Step 12 — Alarms:** Prior steps read what arrived. This step checks what is meant to arrive still can. Per observability doc (health endpoint, alert routing, uptime monitors):
|
|
153
157
|
- Uptime monitors. List each: last check-in, last failure, still right URL/environment. Status-only assertion can pass on login wall when checker follows redirects.
|
|
154
|
-
- Cron coverage. Every clock job health endpoint should know. Unknown job can stop silently.
|
|
158
|
+
- Cron coverage. Every clock job health endpoint should know. Unknown job can stop silently. Read every run, not each job's newest: run started and unfinished past its timeout = finding.
|
|
155
159
|
- Alert routing. Rule exists, destination current, last real alert seen. Alert never seen fire since created = unproven, not healthy. Destination by role or doc section; no personal address.
|
|
156
160
|
- Alert quality. Count fired since previous report, share that led to action, time to acknowledge. Rule fired repeatedly with no action = noise; name it. Rule with no environment filter letting non-production noise reach person = noise; name it.
|
|
157
161
|
- On-call. Who gets production alerts OOH, if no ack, whether quiet hours suppress anything critical. Before launch, developer inbox OK only as written answer.
|
|
158
162
|
|
|
159
|
-
No error tracker → monitor list and alert rules `Not Checked`; cron coverage still from repo. Before launch, much unproven for lack of traffic. Record what could not be proven and why. Missing monitor = next step, not this run. **Done when:** `Alarms` row per monitor and cron: last check-in · what could not be proven. Alert-quality numbers and on-call answer included.
|
|
163
|
+
No error tracker → monitor list and alert rules `Not Checked`; cron coverage still from repo. Before launch, much unproven for lack of traffic. Record what could not be proven and why. Missing monitor = next step, not this run. **Done when:** `Alarms` row per monitor and cron: last check-in · runs left unfinished · what could not be proven. Alert-quality numbers and on-call answer included.
|
|
160
164
|
|
|
161
165
|
- [ ] **Step 13 — Standards:** Confirm current version per anchor standard; read changes since previous report's version. `Standards Coverage`: one row per tracker setup practice (symbolication, release, environment, release health, grouping, ownership, quotas, spike protection, scrubbing) and per trace-propagation and SLO-alerting practice → checked (name step) / not applicable (why) / finding. Unapplied recommended practice → adopt (propose step) / reject (why) / not applicable. **Done when:** no empty row.
|
|
162
166
|
- [ ] **Step 14 — Anything Else:** (a) Everything changed since previous report's commit touching error reporting and logs (tracker init and integrations, scrubber, upstream-outage classifier, new error-level call sites, alert rules, uptime and cron monitors, log-fetch blocks, new outside services) → one-line verdict each. (b) Bounded ~30 min pass: what senior SRE flags that no step asks; read since previous run (incidents, new `docs/failures.jsonl` rows, vendor status incidents) — what would no step catch? Any error source this run saw that `Runtime Logs` does not name → record and propose source in `Process Improvements`. **Done when:** `Beyond the Steps` holds both passes, or `none`. Class seen on 2 runs → propose step in `Process Improvements`.
|
|
@@ -179,7 +183,7 @@ Verify only when applicable.
|
|
|
179
183
|
| `CRITICAL` | Unhandled failure on a production path with no workaround; swallowed write that returns success; message text or credentials stored in the tracker |
|
|
180
184
|
| `HIGH` | Genuine error on a rail with no other coverage; production events above zero and unread; dropped events or quota exhaustion that blinds the board; platform-only failure with no signal; retention shorter than the cadence on a source production depends on |
|
|
181
185
|
| `MEDIUM` | Handled outcome filed at error; unsymbolicated stack; no owner on most unresolved issues; alert that has never fired, or fires with no action; fingerprint explosion; user report with no event |
|
|
182
|
-
| `LOW` | Noise with a named vendor frame; test-traffic issue already tagged;
|
|
186
|
+
| `LOW` | Noise with a named vendor frame; test-traffic issue already tagged; `docs/noise.md` wording drift |
|
|
183
187
|
|
|
184
188
|
Scale per `.bkit/jsonl.md` § Severity. Within a severity: production first, then blast radius. A test-origin tag does not lower severity. A quiet board is not a pass.
|
|
185
189
|
|
|
@@ -244,6 +248,13 @@ PENDING
|
|
|
244
248
|
|
|
245
249
|
PENDING
|
|
246
250
|
|
|
251
|
+
### Suggested Issues
|
|
252
|
+
|
|
253
|
+
<!-- prettier-ignore -->
|
|
254
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
255
|
+
| --- | --- | --- | --- | --- | --- |
|
|
256
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
257
|
+
|
|
247
258
|
### Process Improvements
|
|
248
259
|
|
|
249
260
|
PENDING
|
|
@@ -257,6 +268,7 @@ PENDING
|
|
|
257
268
|
- [ ] **Series:** `Board` counts beside previous report's counts, or marked baseline.
|
|
258
269
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
259
270
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
271
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
260
272
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
261
273
|
|
|
262
274
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -68,10 +68,13 @@ Config ids look identical delisted vs choice day. Target failure: **a stale entr
|
|
|
68
68
|
|
|
69
69
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
70
70
|
|
|
71
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
72
|
+
|
|
71
73
|
#### Core
|
|
72
74
|
|
|
73
75
|
Always verify.
|
|
74
76
|
|
|
77
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
75
78
|
- [ ] **Catalogue Read:** Steps 3, 5, 6 — live model list per provider call sites use, e.g. OpenRouter `/api/v1/models`, Anthropic models API, OpenAI models API, Google models API. Fallback: maker docs only; live-endpoint claims = hypotheses.
|
|
76
79
|
- [ ] **Web:** Steps 5, 10 — maker deprecation pages, model cards, anchor standards. Fallback: versions recorded as unread.
|
|
77
80
|
|
|
@@ -93,7 +96,7 @@ Verify only when applicable.
|
|
|
93
96
|
|
|
94
97
|
### Probe Steps
|
|
95
98
|
|
|
96
|
-
- [ ] **Step 1 — Previous Run:**
|
|
99
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
97
100
|
- [ ] **Step 2 — Detect:** Find every model call site (SDK calls, HTTP to a model API, config lists) and whether choice lives in one registry or literals at sites. Embeddings and other modalities count. Registry absent and ids inline = scatter is the finding; still audit ids. **Done when:** evidence in `Checked`, or `N/A` and STOP.
|
|
98
101
|
- [ ] **Step 3 — Catalogue:** Pull live catalogue for each provider Step 2 found. Record date, URL, counts (full list, any constrained subset calls require, e.g. zero-retention). Every id proposed later must exist in this pull. **Done when:** date, source, and counts in `Catalogue`.
|
|
99
102
|
- [ ] **Step 4 — Call Sites:** Derive each call site's requirements from code this run: modality, tools, and tool choice, structured output, reasoning, output cap, timeout. Do not copy a requirements table forward. Language-server references on each wrapper and each exported chain → every task routing through it; re-run empty results, corroborate with text search. Empty is inconclusive. Task added since previous report appears here only. **Done when:** `Call Sites` row per site: task · wrapper · requirements · chain · `file:line` · how derived.
|
|
@@ -188,6 +191,13 @@ PENDING
|
|
|
188
191
|
|
|
189
192
|
PENDING
|
|
190
193
|
|
|
194
|
+
### Suggested Issues
|
|
195
|
+
|
|
196
|
+
<!-- prettier-ignore -->
|
|
197
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
198
|
+
| --- | --- | --- | --- | --- | --- |
|
|
199
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
200
|
+
|
|
191
201
|
### Process Improvements
|
|
192
202
|
|
|
193
203
|
PENDING
|
|
@@ -201,6 +211,7 @@ PENDING
|
|
|
201
211
|
- [ ] **Series:** every series beside previous report's numbers, or marked baseline.
|
|
202
212
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
203
213
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
214
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
204
215
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
205
216
|
|
|
206
217
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -65,10 +65,13 @@ Privacy rules land with features; nothing re-reads them together. Drift: new tab
|
|
|
65
65
|
|
|
66
66
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
67
67
|
|
|
68
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
69
|
+
|
|
68
70
|
#### Core
|
|
69
71
|
|
|
70
72
|
Always verify.
|
|
71
73
|
|
|
74
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
72
75
|
- [ ] **Published Policy:** Step 10 — privacy policy and in-app privacy copy readable (repo file or live URL). Fallback: unreadable → `Not Checked`; no policy = finding.
|
|
73
76
|
- [ ] **Web:** Steps 2, 6, 12 — regulator and law sources, vendor sub-processor pages, anchor standards. Fallback: regime changes and versions recorded as unread.
|
|
74
77
|
|
|
@@ -89,7 +92,7 @@ Verify only when applicable.
|
|
|
89
92
|
|
|
90
93
|
### Probe Steps
|
|
91
94
|
|
|
92
|
-
- [ ] **Step 1 — Previous Run:**
|
|
95
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
93
96
|
- [ ] **Step 2 — Regimes:** Per candidate regime: applies? why (user locations, turnover / volume thresholds, exemptions and their exceptions, child-directed or actual knowledge). Non-binding regime → mark which checks run as good practice. List law / regulator / code changes since previous report's date, with source links. **Done when:** `Regimes` row per regime: applies · why · changes.
|
|
94
97
|
- [ ] **Step 3 — Inventory:** Schema and code. Per table / column / payload: what, whose (user / non-user member / third party named in content), why held, where documented; strictly necessary? default? Optional on by default or model-prompt data beyond task need = finding. Group by purpose → record-of-processing: lawful basis, subjects, recipients, transfer destination and mechanism, security measure. Consent purposes → consent record and withdrawal path. Third parties in content → legitimate-interests assessment and notify decision. Non-row stores: file storage, device caches, derived data (embeddings as personal as source text). Device storage (cookies, local storage, native caches, SDK identifiers) → strictly necessary vs needs consent. **Done when:** `Inventory` lists every store; store no doc mentions = finding.
|
|
95
98
|
- [ ] **Step 4 — Children:** When product can hold children's data, state every run: how child is known (field / role / parent statement / not at all) and where; consent path per age band and rule setting it; child defaults (visibility, location, sharing, automated processing); age-threshold crossing (re-consent / role change / nothing); parent monitoring of a child (most children's rules require telling the child). **Done when:** answers in `Inventory`, or evidence product cannot hold children's data in `Checked`.
|
|
@@ -189,6 +192,13 @@ PENDING
|
|
|
189
192
|
|
|
190
193
|
PENDING
|
|
191
194
|
|
|
195
|
+
### Suggested Issues
|
|
196
|
+
|
|
197
|
+
<!-- prettier-ignore -->
|
|
198
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
199
|
+
| --- | --- | --- | --- | --- | --- |
|
|
200
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
201
|
+
|
|
192
202
|
### Process Improvements
|
|
193
203
|
|
|
194
204
|
PENDING
|
|
@@ -202,6 +212,7 @@ PENDING
|
|
|
202
212
|
- [ ] **Annual Review:** states whether this run serves as the annual privacy review and whether everyone with children's-data access completed privacy training.
|
|
203
213
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
204
214
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
215
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
205
216
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
206
217
|
|
|
207
218
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -71,10 +71,13 @@ Guides disagree. The guide for the model that runs the prompt wins. A prompt tha
|
|
|
71
71
|
|
|
72
72
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
73
73
|
|
|
74
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
75
|
+
|
|
74
76
|
#### Core
|
|
75
77
|
|
|
76
78
|
Always verify.
|
|
77
79
|
|
|
80
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
78
81
|
- [ ] **Code Search:** Steps 2, 4–9 — text search and language server for call sites (e.g. ripgrep, Serena, `tsserver`). Prompts often markdown or string literals; tool descriptions often strings in schema files. Language-server hits under-report; empty inconclusive → corroborate with text search and name the tool. Fallback: text search only; name it per claim.
|
|
79
82
|
- [ ] **Web:** Steps 3, 10 — each maker's prompting guide (e.g. WebFetch, WebSearch). Fallback: guide practice recorded as unread; Steps 4–7 cite the previous report's guide version and say so.
|
|
80
83
|
|
|
@@ -94,7 +97,7 @@ Verify only when applicable.
|
|
|
94
97
|
|
|
95
98
|
### Probe Steps
|
|
96
99
|
|
|
97
|
-
- [ ] **Step 1 — Previous Run:**
|
|
100
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
98
101
|
- [ ] **Step 2 — Inventory:** Start from code: every model call site outside tests (newest `bkit-audit-llm` `Inventory` as cross-check, not source). Per prompt: file, call sites, model(s) that run it — every model in the fallback chain, resolved at the call site (choice and lifecycle → `bkit-audit-models`, not scored here) · maker guide that applies · task (extract, classify, write, converse, tool-calling) · output consumer (parser, schema, human) · eval set covering it (`bkit-audit-llm` `Evals`) · last-changed commit · model or guide changed since. Exclude bkit-managed files and coding-agent instruction files. Diff vs previous run; new prompt or new model read first. **Done when:** `Inventory` row per prompt, or `N/A` and STOP.
|
|
99
102
|
- [ ] **Step 3 — Guides:** Per maker in `Inventory`: read current prompting guide and model-specific page for each model generation in use. Record URL, last-updated date or date read, generation targeted. Build `Guidance`: practice · source · makers it applies to · which conflict. Conflict between makers → note which prompts run on both. Guide unread → `Not Checked`; Steps 4–7 for that maker = previous report's reading, labelled. **Done when:** `Guidance` row per practice used in Steps 4–7, every row sourced.
|
|
100
103
|
- [ ] **Step 4 — Clarity:** Per prompt: task and what a good answer looks like stated up front; audience and context given; each constraint carries its reason where the guide asks for one; says what to do, not only what to avoid; terms the output depends on defined (e.g. date format, "recent", allowed labels); no contradiction within the prompt or between system prompt and template; no dead instruction naming a removed field, tool, or format. **Done when:** `Clarity` row per prompt: verdict per check, each failure cited.
|
|
@@ -193,6 +196,13 @@ PENDING
|
|
|
193
196
|
|
|
194
197
|
PENDING
|
|
195
198
|
|
|
199
|
+
### Suggested Issues
|
|
200
|
+
|
|
201
|
+
<!-- prettier-ignore -->
|
|
202
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
203
|
+
| --- | --- | --- | --- | --- | --- |
|
|
204
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
205
|
+
|
|
196
206
|
### Process Improvements
|
|
197
207
|
|
|
198
208
|
PENDING
|
|
@@ -207,6 +217,7 @@ PENDING
|
|
|
207
217
|
- [ ] **Guides:** every `Model Fit` verdict names the guide and date it was judged against.
|
|
208
218
|
- [ ] **Writes:** only this report and template `Project Instructions` changed.
|
|
209
219
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
220
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
210
221
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
211
222
|
|
|
212
223
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -79,10 +79,13 @@ One per probe, on the `Probes` row.
|
|
|
79
79
|
|
|
80
80
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
81
81
|
|
|
82
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
83
|
+
|
|
82
84
|
#### Core
|
|
83
85
|
|
|
84
86
|
Always verify.
|
|
85
87
|
|
|
88
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
86
89
|
- [ ] **Target:** Steps 2–12 — non-production deployment of this commit, resolved by commit SHA, confirmed serving. Fallback: none. Probes stay `INCONCLUSIVE`. NEVER point them at production.
|
|
87
90
|
- [ ] **Identities:** Steps 2, 5–12, 14 — create and delete synthetic users and tenants, guarded so a non-synthetic identity cannot be deleted. Fallback: none. Do not probe with real accounts.
|
|
88
91
|
- [ ] **Web:** Steps 2, 13 — each platform's penetration-testing page, and anchor standards. Fallback: policy unread → do not send probes; record why.
|
|
@@ -106,7 +109,7 @@ Verify only when applicable.
|
|
|
106
109
|
|
|
107
110
|
### Probe Steps
|
|
108
111
|
|
|
109
|
-
- [ ] **Step 1 — Previous Run:**
|
|
112
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
110
113
|
- [ ] **Step 2 — Engage:** Record commit and non-production URL deployed for it (the deployment the live lane resolves for that SHA, not a branch alias that may be stale). Confirm serving. Write scope in `Scope`. **In:** non-production web URL (e.g. preview deploy, staging), API host, machine-client endpoint, inbound mail domain, data API via client public key, background-job serve endpoint, native build if tested. **Out:** production, release branch, persistent demo tenant, every third party's own platform, anything not on in-list. Read current penetration-testing page of each platform probes touch (app host, database, native vendor if in scope). Record what they allow and plan tier. Page forbids probe → do not send; `INCONCLUSIVE`. Create two synthetic tenants, attacker, and victim, each with roles the product distinguishes. Issue machine-client token for each owner where product has one, on plan the feature allows. Seed victim with few records, one recurring record, attachment. Do not touch persistent demo tenant. Read newest `bkit-audit-security` and `bkit-audit-llm` reports. Every finding they could not prove → probe; model audit's prompt-only rules → probe. Every trust boundary threat model names → ≥1 probe. No threat model → say so in `Scope`. Before destructive probe, confirm no other live test holds target database. Record window. Do not wait for named approver. Later probe degrades target, touches non-synthetic data, or reaches third party → stop remaining probes; record stop in `Scope`. **Done when:** `Scope` holds URL, commit, in-list, out-list, platform rules, plan tier, identities (no tokens), suspicion list. **Probe row** (every later probe): request with tokens redacted, response status, relevant body, verdict. `BROKE` adds blast radius by reasoning, not pivoting.
|
|
111
114
|
- [ ] **Step 3 — Surface:** List attack surface at this commit: routes, server actions, machine-client tools, webhooks, response headers of each in-scope origin. Diff vs previous report's list. Everything new → ≥1 probe. **Done when:** `Surface` holds list and diff.
|
|
112
115
|
- [ ] **Step 4 — Retest:** Re-run every earlier `BROKE` whose fix landed, from recorded reproduction. `FIXED` or `STILL BROKE`, with fix commit. Confirm regression test prior run named exists; if not → `bkit-audit-testing`. **Done when:** `Retests` row per prior break, or `none`.
|
|
@@ -119,7 +122,7 @@ Verify only when applicable.
|
|
|
119
122
|
- [ ] **Step 11 — Tools:** With attacker's token, as client that ignores rules: set fields allow-list excludes; pass victim's ids; change recurring record without required scope; send oversized arguments; repeat create to test idempotency key. **Done when:** `Probes` row per case, or skip with evidence no machine-client tools.
|
|
120
123
|
- [ ] **Step 12 — Boundaries:** Script and markup in text fields; check render on web, native client, reply. Injection through search query and filters reaching search routine. Webhook endpoints: no signature = denial; wrong signature = same denial plus error log. Rate limits: documented thresholds fire, then stop. **Done when:** `Probes` row per boundary.
|
|
121
124
|
- [ ] **Step 13 — Standards:** Confirm current version per anchor standard; read changes since previous report's version. `Standards Coverage`: one row per WSTG category (INFO, CONF, IDNT, ATHN, ATHZ, SESS, INPV, ERRH, CRYP, BUSL, CLNT, APIT), and per ATLAS tactic a model probe touched → checked (name step) / not applicable (why) / finding. Unapplied recommended practice → adopt (propose step) / reject (why) / not applicable. **Done when:** no empty row.
|
|
122
|
-
- [ ] **Step 14 — Anything Else:** (a) Everything changed since previous report's commit touching attack surface (routes, server actions, machine-client tools, webhooks, response headers, inbound handling, what client public key can reach; Step 3 diff = starting list) → one-line verdict each. (b) Bounded ~30 min pass: what senior penetration tester flags that no step asks; read since previous run (incidents, `bkit-audit-logs` reports, new `docs/failures.jsonl` rows, vendor status incidents) — what would no step catch? Probes these passes send follow rules, use identities made for it, go in `Probes`. Then delete every identity this run created with guarded delete (reaps tenant it would strand); confirm orphan sweep finds nothing left. Record where full request/response evidence is kept, for how long, redaction applied. **Done when:** `Beyond the Steps` holds both passes, or `none`; `Cleanup` confirms every identity this run created is gone. Class seen on 2 runs → propose step in `Process Improvements`. `Expected` names where finding goes, recommends safer fix. This run does not file
|
|
125
|
+
- [ ] **Step 14 — Anything Else:** (a) Everything changed since previous report's commit touching attack surface (routes, server actions, machine-client tools, webhooks, response headers, inbound handling, what client public key can reach; Step 3 diff = starting list) → one-line verdict each. (b) Bounded ~30 min pass: what senior penetration tester flags that no step asks; read since previous run (incidents, `bkit-audit-logs` reports, new `docs/failures.jsonl` rows, vendor status incidents) — what would no step catch? Probes these passes send follow rules, use identities made for it, go in `Probes`. Then delete every identity this run created with guarded delete (reaps tenant it would strand); confirm orphan sweep finds nothing left. Record where full request/response evidence is kept, for how long, redaction applied. **Done when:** `Beyond the Steps` holds both passes, or `none`; `Cleanup` confirms every identity this run created is gone. Class seen on 2 runs → propose step in `Process Improvements`. `Expected` names where finding goes, recommends safer fix. This run does not file; the audit agent writes `docs/failures.jsonl` rows: `BROKE` → plan from this report, `docs/failures.jsonl` row, regression test (ideally deployed boundary) so `bkit-audit-testing` keeps it fixed; `HELD` with no test → `bkit-audit-testing`; model result → `bkit-audit-llm`, against finding it tested; `STILL BROKE` → original finding, with retest and fix commit; `INCONCLUSIVE` → next run's Step 2 suspicion list, with what blocked; `OTHER` → report, second run proposes verdict; vulnerability in database product itself, not project's use → vendor per that vendor's testing page, plus this report; stop condition → remaining probes stopped, stop recorded in `Scope`.
|
|
123
126
|
|
|
124
127
|
### 🔒 Gate 2: Probes Check
|
|
125
128
|
|
|
@@ -198,6 +201,13 @@ PENDING
|
|
|
198
201
|
|
|
199
202
|
PENDING
|
|
200
203
|
|
|
204
|
+
### Suggested Issues
|
|
205
|
+
|
|
206
|
+
<!-- prettier-ignore -->
|
|
207
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
208
|
+
| --- | --- | --- | --- | --- | --- |
|
|
209
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
210
|
+
|
|
201
211
|
### Process Improvements
|
|
202
212
|
|
|
203
213
|
PENDING
|
|
@@ -211,6 +221,7 @@ PENDING
|
|
|
211
221
|
- [ ] **Cleanup:** states that every identity this run created is deleted, and where redacted evidence is kept.
|
|
212
222
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
213
223
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
224
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
214
225
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
215
226
|
|
|
216
227
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -71,10 +71,13 @@ Failure paths are least-tested: happy path is tested; vendor outage stays unwalk
|
|
|
71
71
|
|
|
72
72
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
73
73
|
|
|
74
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
75
|
+
|
|
74
76
|
#### Core
|
|
75
77
|
|
|
76
78
|
Always verify.
|
|
77
79
|
|
|
80
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
78
81
|
- [ ] **Code Search:** Steps 2–12, 14, 16 — text search and language server for callers, e.g. ripgrep, Serena, `tsserver`. Language-server hits under-report; empty is inconclusive → corroborate with text search and name the tool. Fallback: text search only; name it per claim.
|
|
79
82
|
- [ ] **Web:** Step 15 — fetch anchor standards. Fallback: versions recorded as unread.
|
|
80
83
|
|
|
@@ -83,7 +86,7 @@ Always verify.
|
|
|
83
86
|
Verify only when applicable.
|
|
84
87
|
|
|
85
88
|
- [ ] **Metrics:** Hosted traces, errors, or health endpoint → Steps 3, 13 — SLO queries and alert config, e.g. Sentry, Datadog, Better Stack. Fallback: repo config only; live measurements `Not Checked`.
|
|
86
|
-
- [ ] **Job Platform:** Background jobs → Steps 6, 7 — flow control and replay docs, read-only, e.g. Inngest, Temporal. Fallback: code only; live run history `Not Checked`.
|
|
89
|
+
- [ ] **Job Platform:** Background jobs → Steps 6, 7 — flow control and replay docs, read-only, e.g. Inngest, Temporal. Log access: `.agents/context.md` `context-fetch` blocks below `bkit:end` are ready-made read-only readers (`bkit context list-blocks` → each block's `context` and `body`); set a body's `# Customise:` knobs to the audit window; run it via `direnv exec . bash`. Fallback: code only; live run history `Not Checked`.
|
|
87
90
|
- [ ] **Deploy Platform:** Hosted deploys → Step 12 — skew protection, rollout, rollback history, e.g. Vercel API, Cloud Run. Fallback: repo config only; dashboard settings `Not Checked`.
|
|
88
91
|
- [ ] **Vendor Status:** External dependencies → Step 13 — status feeds. Fallback: none; feeds `Not Checked`.
|
|
89
92
|
|
|
@@ -96,7 +99,7 @@ Verify only when applicable.
|
|
|
96
99
|
|
|
97
100
|
### Probe Steps
|
|
98
101
|
|
|
99
|
-
- [ ] **Step 1 — Previous Run:**
|
|
102
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
100
103
|
- [ ] **Step 2 — Surfaces:** One table, kept run to run for diffs. Outbound calls (every SDK client and every `fetch` host); webhook receivers; background work (job functions and scheduled database jobs); write paths (routes and server actions that change data). Per dependency: hard (journey fails without it) or soft (degrades), and what it depends on in turn. Transitive counts (e.g. auth mail via mail vendor → mail outage is sign-in outage). Surface in code, not docs = finding; doc rewrite → `bkit-audit-documentation`. **Done when:** `Surfaces` lists every surface with hard/soft and transitive deps, or `N/A` and STOP.
|
|
101
104
|
- [ ] **Step 3 — Journeys:** Every user-reliant journey. Per journey: SLI as good/total ratio, SLO over rolling 28-day window, current measurement with query that produced it. Keep table run to run. No target, or target nobody measures = finding. Error-budget policy: what changes when budget spent (e.g. feature freeze, mandatory postmortem when one incident uses >20% of budget). Record whether policy triggered since previous run and whether obeyed. No policy = finding. **Done when:** `Journeys` row per journey: SLI · SLO · measurement and query · budget triggered · obeyed.
|
|
102
105
|
- [ ] **Step 4 — Dropped Errors:** Codebase-wide, not only errors that reached a log (`bkit-audit-logs` owns those). `catch` that neither rethrows, returns error, nor logs at `warn` or above; client calls whose error never read (e.g. Supabase `{ error }`, AWS SDK, Prisma); promises without `await` or `.catch`; broad catches turning every failure into one generic message so caller cannot tell timeout from bug. Dropped, never logged = invisible to user and operators; those first. Deliberate best-effort (telemetry, fire-and-forget notification) OK when code says so. Linter-catchable class (e.g. `@typescript-eslint/no-floating-promises`, unread client `error`) → name rule under `Process Improvements`; do not add it. Adding gate → `bkit-audit-gates`. **Done when:** every hit = finding with `file:line`.
|
|
@@ -199,6 +202,13 @@ PENDING
|
|
|
199
202
|
|
|
200
203
|
PENDING
|
|
201
204
|
|
|
205
|
+
### Suggested Issues
|
|
206
|
+
|
|
207
|
+
<!-- prettier-ignore -->
|
|
208
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
209
|
+
| --- | --- | --- | --- | --- | --- |
|
|
210
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
211
|
+
|
|
202
212
|
### Process Improvements
|
|
203
213
|
|
|
204
214
|
PENDING
|
|
@@ -213,6 +223,7 @@ PENDING
|
|
|
213
223
|
- [ ] **Game Day:** states the date and outcome of the last game day, or that none is recorded, and names the next dependency.
|
|
214
224
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
215
225
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
226
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
216
227
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
217
228
|
|
|
218
229
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. `@bkit-audit-generate` fills this template. -->
|
|
1
|
+
<!-- Note: Reports live at `docs/audits/{YYYYMM}/{slug}.md`. `{YYYYMM}` = UTC month of this run. `{slug}` = template basename without `bkit-audit-` and `.md`. **If** that month already has `{slug}.md` or `{slug}-{TIMESTAMP}.md` → write `{slug}-{TIMESTAMP}.md` (`{TIMESTAMP}` = `date -u +%Y%m%dT%H%M%SZ`). Never overwrite. The report is stamped with a `bkit:anchor` block recording the audited commit (`baseSha`) and branch (`baseRef`); never hand-author or edit it. `@bkit-audit-generate` fills this template. -->
|
|
2
2
|
|
|
3
3
|
<!-- bkit:start -->
|
|
4
4
|
|
|
@@ -81,10 +81,13 @@ Every task leaves a worktree, branch, stash, or scratch directory; nothing's job
|
|
|
81
81
|
|
|
82
82
|
**Checkpoint:** Check off all core items. Check conditional items only when applicable. Each item: available → tool and version; missing → fallback, gap in `Not Checked`, fix in `Process Improvements`. Unnamed gap reads as passed. never abort for a missing capability.
|
|
83
83
|
|
|
84
|
+
**Credentials:** probe direnv before marking a token / env var missing: `direnv exec . sh -c '[ -n "${VAR}" ] && echo set || echo unset'`; `unset` = missing. Run credentialed CLIs as `direnv exec . <cmd>`. No direnv (cloud session) → plain env check. Report names, never values.
|
|
85
|
+
|
|
84
86
|
#### Core
|
|
85
87
|
|
|
86
88
|
Always verify.
|
|
87
89
|
|
|
90
|
+
- [ ] **Permissions:** Before Step 1 — one cheap read-only call per tool this run uses (every item below, every rule under `Project Instructions` `Permissions`), e.g. `gh run list --limit 1`, one MCP read, one anchor-standard fetch. Blocked / denied (here or in any step) → record call verbatim and narrow allow rule clearing it (e.g. `Bash(gh run list *)`) in `Not Checked`, `Process Improvements`, and `Project Instructions` `Permissions`; run every step the remaining tools allow; work needing blocked call → `Not Checked`. never stop the run for a blocked call; never retry or reword a blocked call to get past it.
|
|
88
91
|
- [ ] **Git:** Steps 2–7, 9 — local checkout, worktrees, branches, stashes. Fallback: none; no checkout → `N/A` and STOP.
|
|
89
92
|
- [ ] **Host API:** Steps 4, 8 — read-only, e.g. `gh api`. Fallback: local git only; remote branches, pull requests, and settings `Not Checked`.
|
|
90
93
|
- [ ] **Web:** Step 10 — fetch anchor standards. Fallback: versions recorded as unread.
|
|
@@ -107,7 +110,7 @@ Verify only when applicable.
|
|
|
107
110
|
|
|
108
111
|
### Probe Steps
|
|
109
112
|
|
|
110
|
-
- [ ] **Step 1 — Previous Run:**
|
|
113
|
+
- [ ] **Step 1 — Previous Run:** Audited commit = this report's `bkit:anchor` `baseSha`. Read newest prior report for this slug under `docs/audits/`; its commit = its `bkit:anchor` `baseSha`, else the HEAD its `Previous Run` recorded. Count findings fixed / still open; age of oldest open. Open after 2 runs → re-raise with age. **Done when:** closure rate in `Previous Run`, or run marked baseline.
|
|
111
114
|
- [ ] **Step 2 — Checkout:** Name machine and checkout. Worktrees, stashes, scratch directories are local — fresh cloud checkout sees none; say so, not clean. Remote branches and pull requests are shared; every run checks those. **Done when:** machine, checkout, and which surfaces are local-only in `Checked`.
|
|
112
115
|
- [ ] **Step 3 — Worktrees:** `git worktree list --porcelain`. Per worktree: branch, ahead/behind integration branch, `git cherry`, pull-request state, uncommitted changes (`git status --porcelain`), last commit date, size (`du -sh`). Locked or files modified in the last day → KEEP or ASK, never REMOVE (agent may still be running). Also `git worktree prune --dry-run` for entries whose directory is gone. Never prune for real. **Done when:** `Worktrees` row per entry with verdict; totals in `Totals`.
|
|
113
116
|
- [ ] **Step 4 — Branches:** Local: merged by ancestry, merged by squash (pull request merged, or `git cherry` all `-`), upstream gone, agent-prefix branches with no worktree. Remote: branches other than integration, production, dependency-update bot's, with pull-request state. Record delete-on-merge; when off, merged pull request leaves its branch. Unmerged with no pull request: every commit is RESCUE until shown otherwise. Open pull requests with no activity for 30 days, and old drafts. Tags no release or deploy uses. **Done when:** `Branches` row per item with verdict; counts in `Totals`.
|
|
@@ -115,7 +118,7 @@ Verify only when applicable.
|
|
|
115
118
|
- [ ] **Step 6 — Leftovers:** Untracked (`git status --porcelain`) and ignored (`git ls-files --others --ignored --exclude-standard --directory`), each sized. Sort: regenerable (dependencies, task cache, coverage, test results, native build folders, tool caches) — REMOVE is safe, report the size; hand-made (notes, plans, backups, logs, scripts in the scratch directory) — group by age and type; open issue or plan references → KEEP; produced and not cleaned up — leftover REMOVE, producer is finding; stranded at root — find writer; tool defaulting to root → point elsewhere. Ignore file: rules matching nothing; leftovers untracked because no rule covers them. Scan secrets and personal data before any MOVE or REMOVE on scratch, root, or plan logs; scanner reports outside repo. Personal-data hits are candidates: test fixtures carry fake addresses. A real hit in a tracked file is not fixed by removing the file — it stays in history. Name the file and line, never the value. Hooks: hooks path resolves at the hook directory, each hook executable, and the install script installs them. A checkout where hooks silently do not run looks clean here while skipping every pre-commit control. **Done when:** `Leftovers` grouped with verdicts; scan result named; sizes in `Totals`.
|
|
116
119
|
- [ ] **Step 7 — Layout:** Read where tracked files belong; find misfits. Size: largest blobs at `HEAD`. Committed generated output/logs → MOVE or REMOVE candidates. Record pack size as series. History size: deleted file still grows clones — repository sizer or blob sizes across all refs. Images/logs → recommend large-file storage or ignore; history rewrite owner's call (ASK). Secrets in history: scan all refs; list every sensitive file ever added (example env files expected). Verified live credential → `CRITICAL`: rotation first (rewrite does not reach existing clones). Name file and commit, never value. Toolchain agreement: runtime pin, engine field, package-manager field, CI setup, host build image, native build config; workspace dependency-version linter for cross-workspace drift or catalog bypass. Docs-versus-files → documentation audit; this step files versus each other. Accidental publish: every workspace package not meant for a registry is marked unpublishable (e.g. npm `private: true`, Cargo `publish = false`), and the root states the licence position. Attributes: end-of-line, generated-file linguist marks so review diffs collapse them, binary marks on images, and an editor config that agrees with the formatter. Placement: root files belonging to app/package, finished one-off scripts, drafts beside final doc, unlinked archive folders, near-empty packages. Naming: folders/files breaking neighbour conventions. Contribution scaffolding: owners file, pull-request and issue templates, contributing note — present and true; whether team needs them → owner (ASK). Template describing unfollowed process → MOVE or REMOVE. Owners file: propose paths that weaken a gate when edited → required human review; once exists, host owners-errors API must return none. Unimported file → dead-code audit; misplaced-but-wrong doc → documentation audit. **Done when:** `Layout` rows; pack size and largest history blobs in `Totals`.
|
|
117
120
|
- [ ] **Step 8 — Posture:** Config audit asks settings vs docs; this step asks vs anchor standards. Read that audit's latest report first; do not duplicate findings. Shared settings: every run checks (local or cloud). Scorecard: each check's score as series. Keep binary artifacts, branch protection, code review, dependency-update tool, security policy here; token permissions, pinned dependencies, dangerous workflow → `bkit-audit-security`. Branch protection (integration, production): pull request required, required check required, force-push and deletion blocked, squash-only or linear history, bypass list — compare with production-merge-requires-PR rule. Security settings: secret scanning and push protection on; who may bypass push protection. Private repo: push protection may need paid secret-protection plan — say so if plan excludes it. Push protection blocks at server (hook bypass does not reach). Also dependency alerts, private vulnerability reporting, default automation token read-only, automation creating/approving pull requests off. Disclosure: security policy or private vulnerability reporting, and `/.well-known/security.txt` on production web origin with RFC contact/expiry fields and expiry ahead. People, 2FA, deploy keys, long-lived tokens → security audit; automation secrets/variables → config audit. Read their latest reports; confirm 2FA enforcement recorded. List accumulators like branches: installed apps and permissions, webhooks, environments. App over-scoped or webhook to unused service → REMOVE or ASK. **Done when:** `Posture` holds each setting met / finding / not applicable; scores in `Totals`.
|
|
118
|
-
- [ ] **Step 9 — Verify:** Git state changes hourly. Record when each check ran; re-check every RESCUE and REMOVE at end. No finding without re-checkable evidence: command and output, or `file:line`. Destinations (name; do not act): REMOVE → exact command next step would run, batched for one review; RESCUE → what it holds, who would know, decision (push and open pull request, or drop on purpose); MOVE → move and every reference to update; ASK → who would know and question; OTHER → why no verdict fits; leftover something keeps producing → `docs/failures.jsonl` row against producer
|
|
121
|
+
- [ ] **Step 9 — Verify:** Git state changes hourly. Record when each check ran; re-check every RESCUE and REMOVE at end. No finding without re-checkable evidence: command and output, or `file:line`. Destinations (name; do not act; the audit agent writes `docs/failures.jsonl` rows): REMOVE → exact command next step would run, batched for one review; RESCUE → what it holds, who would know, decision (push and open pull request, or drop on purpose); MOVE → move and every reference to update; ASK → who would know and question; OTHER → why no verdict fits; leftover something keeps producing → `docs/failures.jsonl` row against producer; missing/stale ignore rule → line to add or remove; code or config nothing reaches → `bkit-audit-dead-code`; a doc that is wrong, not misplaced → `bkit-audit-documentation`; dependency-bot branch → `bkit-audit-dependencies`; secret in history, log, or leftover → `bkit-audit-security`, rotation first, file and commit only; personal data in log or leftover → `bkit-audit-privacy`; repository setting below standard → setting to change for owner, doc drift → `bkit-audit-config`; workflow-level scorecard failure → `bkit-audit-security`; toolchain versions disagree → file to change; host setting → `bkit-audit-config`; dependency versions → `bkit-audit-dependencies`. **Done when:** every RESCUE and REMOVE re-checked, with time, verdict, and destination.
|
|
119
122
|
- [ ] **Step 10 — Standards:** Confirm current version per anchor standard; read changes since previous report's version. `Standards Coverage`: one row per Scorecard check and per baseline control in sets this audit touches (e.g. OSPS-AC, OSPS-BR-07, OSPS-GV, OSPS-LE, OSPS-VM) → checked (name step) / not applicable (why) / finding. Unapplied recommended practice → adopt (propose step) / reject (why) / not applicable. **Done when:** no empty row.
|
|
120
123
|
- [ ] **Step 11 — Anything Else:** (a) Everything changed since previous report's commit touching the repository itself (new top-level files and folders, workspaces, ignore, and attribute files, hooks, branch, and ruleset settings, host apps, webhooks, environments, anything new that writes to the scratch directory or the root) → one-line verdict each. (b) Bounded ~30 min pass: what a senior platform engineer flags that no step asks; read what happened since previous run (incidents, `bkit-audit-logs` reports, new `docs/failures.jsonl` rows, vendor status incidents) — what would no step catch? **Done when:** `Beyond the Steps` holds both passes, or `none`. Class seen on 2 runs → propose step in `Process Improvements`.
|
|
121
124
|
|
|
@@ -202,6 +205,13 @@ PENDING
|
|
|
202
205
|
|
|
203
206
|
PENDING
|
|
204
207
|
|
|
208
|
+
### Suggested Issues
|
|
209
|
+
|
|
210
|
+
<!-- prettier-ignore -->
|
|
211
|
+
| # | Title | Findings | Severity | Next Step | Reason |
|
|
212
|
+
| --- | --- | --- | --- | --- | --- |
|
|
213
|
+
| PENDING | PENDING | PENDING | PENDING | PENDING | PENDING |
|
|
214
|
+
|
|
205
215
|
### Process Improvements
|
|
206
216
|
|
|
207
217
|
PENDING
|
|
@@ -215,6 +225,7 @@ PENDING
|
|
|
215
225
|
- [ ] **Series:** `Totals` beside previous report's numbers, or marked baseline.
|
|
216
226
|
- [ ] **Writes:** only this report and template `Project Instructions` changed; throwaway worktrees / remote targets removed.
|
|
217
227
|
- [ ] **Process Improvements:** `Universal` = proposed step changes, template untouched; `Project` = edits applied below `bkit:end`; each `none` if empty.
|
|
228
|
+
- [ ] **Suggested Issues:** every `MEDIUM`+ finding ID in exactly one row; every row has a `Next Step`.
|
|
218
229
|
- [ ] **Gate 3 - Verified:** Every checkbox above independently verified and checked off.
|
|
219
230
|
|
|
220
231
|
> `Project Instructions` below `bkit:end` are consumer-owned — `bkit install` never overwrites them, `bkit check` drift stays clean. Add project-specific tools, paths, commands, and checks there, keyed by section or step title; never results.
|