mandrel 2.101.0 → 2.102.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/audit-checklists/resilience.md +40 -0
- package/.agents/audit-checklists/sre.md +2 -4
- package/.agents/docs/workflows.md +2 -1
- package/.agents/schemas/audit-rules.json +44 -2
- package/.agents/schemas/audit-rules.schema.json +2 -2
- package/.agents/scripts/lib/audit-suite/selector.js +80 -1
- package/.agents/scripts/lib/audit-to-stories/audit-lenses.js +1 -0
- package/.agents/workflows/audit-resilience.md +290 -0
- package/.agents/workflows/audit-sre.md +12 -21
- package/docs/CHANGELOG.md +7 -0
- package/package.json +1 -1
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
<!-- GENERATED FILE — do not edit by hand.
|
|
2
|
+
Source of truth: .agents/workflows/audit-resilience.md
|
|
3
|
+
Regenerate: node .agents/scripts/generate-lens-checklists.js
|
|
4
|
+
Drift is gated by: npm run docs:check
|
|
5
|
+
-->
|
|
6
|
+
|
|
7
|
+
# Resilience Audit — authoring checklist
|
|
8
|
+
|
|
9
|
+
> Audit every external dependency, platform binding and cross-service seam: whether failure is bounded, whether the system degrades on purpose, what the user sees, and whether a human is told and how fast. A deliberately-global lens that builds a dependency inventory and follows…
|
|
10
|
+
|
|
11
|
+
Self-check your change against this lens's concerns before you ship:
|
|
12
|
+
|
|
13
|
+
- [ ] Timeout scope
|
|
14
|
+
- [ ] Retry safety
|
|
15
|
+
- [ ] Outage as empty
|
|
16
|
+
- [ ] Fail-open vs fail-closed
|
|
17
|
+
- [ ] Missing secret or binding
|
|
18
|
+
- [ ] Bypassed primitive
|
|
19
|
+
- [ ] Idempotency keys
|
|
20
|
+
- [ ] Claim before transmit
|
|
21
|
+
- [ ] Webhook receivers
|
|
22
|
+
- [ ] Reconciliation coverage
|
|
23
|
+
- [ ] Compensation
|
|
24
|
+
- [ ] Fan-out settlement
|
|
25
|
+
- [ ] Sink classification
|
|
26
|
+
- [ ] Capture census
|
|
27
|
+
- [ ] Storms and collapse
|
|
28
|
+
- [ ] Silent producers
|
|
29
|
+
- [ ] Sink posture
|
|
30
|
+
- [ ] Watch the watcher
|
|
31
|
+
- [ ] Error page
|
|
32
|
+
- [ ] Outage shape
|
|
33
|
+
- [ ] Retry affordance
|
|
34
|
+
- [ ] Error boundaries
|
|
35
|
+
- [ ] Report path
|
|
36
|
+
- [ ] Per-job isolation
|
|
37
|
+
- [ ] Per-job deadline
|
|
38
|
+
- [ ] Overlap safety
|
|
39
|
+
- [ ] Heartbeats
|
|
40
|
+
- [ ] …see the full lens for the remaining concerns
|
|
@@ -10,17 +10,15 @@
|
|
|
10
10
|
|
|
11
11
|
Self-check your change against this lens's concerns before you ship:
|
|
12
12
|
|
|
13
|
-
- [ ] Run the
|
|
13
|
+
- [ ] Run the readiness battery.
|
|
14
14
|
- [ ] Rollback Path
|
|
15
15
|
- [ ] Migration Reversibility
|
|
16
16
|
- [ ] Feature-Flag Kill Switch
|
|
17
17
|
- [ ] Structured Logging
|
|
18
18
|
- [ ] Metrics & Tracing
|
|
19
19
|
- [ ] Alerting & SLOs
|
|
20
|
-
- [ ] Timeouts & Cancellation
|
|
21
|
-
- [ ] Retry & Backoff
|
|
22
20
|
- [ ] Graceful Shutdown
|
|
23
|
-
- [ ]
|
|
21
|
+
- [ ] Dependency barriers
|
|
24
22
|
- [ ] Runbook Coverage
|
|
25
23
|
- [ ] Health & Readiness
|
|
26
24
|
- [ ] On-Call Escalation
|
|
@@ -51,7 +51,7 @@ by `node .agents/scripts/generate-workflows-doc.js`; `npm run docs:check`
|
|
|
51
51
|
fails when it drifts from the on-disk workflow set. To change a command’s
|
|
52
52
|
description, edit the workflow file’s front-matter and regenerate.
|
|
53
53
|
|
|
54
|
-
## Commands (
|
|
54
|
+
## Commands (37)
|
|
55
55
|
|
|
56
56
|
| Command | Description |
|
|
57
57
|
| --- | --- |
|
|
@@ -73,6 +73,7 @@ description, edit the workflow file’s front-matter and regenerate.
|
|
|
73
73
|
| `/audit-privacy` | Audit logs, telemetry, and persistence paths for PII leakage and retention violations; surface secrets exposure and consent gaps. |
|
|
74
74
|
| `/audit-quality` | Audit test coverage gaps, flaky tests, missing assertions, and test-pyramid balance; recommend a remediation batch. |
|
|
75
75
|
| `/audit-red-team` | Audit the project as a goal-directed adversary — attack paths that chain two or more crossings, abuse of intended features, agent exposure to hostile repository or inbound content, hostile operations and operator hazards — graded under a reproduced-or-reasoned evidence bar. A deliberately-global lens that runs on any project. |
|
|
76
|
+
| `/audit-resilience` | Audit every external dependency, platform binding and cross-service seam: whether failure is bounded, whether the system degrades on purpose, what the user sees, and whether a human is told and how fast. A deliberately-global lens that builds a dependency inventory and follows each failure to a human. |
|
|
76
77
|
| `/audit-security` | Audit dependency CVEs, input-validation gaps, secrets handling, auth boundaries, and the trust-boundary attack classes the baseline is silent on (business logic, SSRF/traversal, deserialization, crypto misuse, feature abuse, LLM tool-binding); emit a structured High/Medium/Low findings report. |
|
|
77
78
|
| `/audit-seo` | Audit SEO fundamentals and Generative Engine Optimization signals (meta, structured data, crawlability); only relevant for web targets. |
|
|
78
79
|
| `/audit-sre` | "Audit production-readiness for a release candidate: SLOs, observability, runbooks, error budgets, and rollback paths." |
|
|
@@ -196,8 +196,7 @@
|
|
|
196
196
|
"rollback",
|
|
197
197
|
"error budget",
|
|
198
198
|
"on-call",
|
|
199
|
-
"incident"
|
|
200
|
-
"resilience"
|
|
199
|
+
"incident"
|
|
201
200
|
],
|
|
202
201
|
"filePatterns": [
|
|
203
202
|
"**/Dockerfile",
|
|
@@ -482,6 +481,49 @@
|
|
|
482
481
|
},
|
|
483
482
|
"scope": "global",
|
|
484
483
|
"substitutionKeys": []
|
|
484
|
+
},
|
|
485
|
+
"audit-resilience": {
|
|
486
|
+
"triggers": {
|
|
487
|
+
"gates": ["gate1", "gate3"],
|
|
488
|
+
"keywords": [
|
|
489
|
+
"resilience",
|
|
490
|
+
"graceful degradation",
|
|
491
|
+
"fallback",
|
|
492
|
+
"timeout",
|
|
493
|
+
"retry",
|
|
494
|
+
"backoff",
|
|
495
|
+
"idempotency",
|
|
496
|
+
"idempotent",
|
|
497
|
+
"circuit breaker",
|
|
498
|
+
"outage",
|
|
499
|
+
"upstream",
|
|
500
|
+
"webhook",
|
|
501
|
+
"reconcile",
|
|
502
|
+
"reconciliation",
|
|
503
|
+
"dead letter",
|
|
504
|
+
"alerting",
|
|
505
|
+
"pager",
|
|
506
|
+
"on-call"
|
|
507
|
+
],
|
|
508
|
+
"filePatterns": [
|
|
509
|
+
"**/webhooks/**",
|
|
510
|
+
"**/webhook/**",
|
|
511
|
+
"**/integrations/**",
|
|
512
|
+
"**/adapters/**",
|
|
513
|
+
"**/providers/**",
|
|
514
|
+
"**/*outbox*",
|
|
515
|
+
"**/*retry*",
|
|
516
|
+
"**/*queue*",
|
|
517
|
+
"**/wrangler.toml",
|
|
518
|
+
"**/wrangler.json",
|
|
519
|
+
"**/wrangler.jsonc",
|
|
520
|
+
"**/serverless.yml",
|
|
521
|
+
"**/serverless.yaml"
|
|
522
|
+
]
|
|
523
|
+
},
|
|
524
|
+
"target": "integrations",
|
|
525
|
+
"scope": "global",
|
|
526
|
+
"substitutionKeys": []
|
|
485
527
|
}
|
|
486
528
|
},
|
|
487
529
|
"sensitivePaths": {
|
|
@@ -61,8 +61,8 @@
|
|
|
61
61
|
},
|
|
62
62
|
"target": {
|
|
63
63
|
"type": "string",
|
|
64
|
-
"enum": ["web", "data-model", "policies", "any"],
|
|
65
|
-
"description": "Target-applicability gate: the kind of project this lens can say anything about. `web` — the lens reads a rendered web surface (pages, routes, HTML/CSS, design tokens) and is skipped entirely on a project that has none, however well its keyword or file-pattern triggers match. `data-model` — the lens reads a persistence layer (ORM models, schema migrations, seed data) and is skipped entirely on a project that has none. `policies` — the lens reads a published customer-facing policy corpus (Terms, Privacy, Cookie, refund/cancellation, consent and waiver text) and is skipped entirely on a project that has none. `any` — always applicable. ABSENT MEANS `any`: omitting the key never changes a lens's behaviour. Applicability is decided by a probe in lib/audit-suite/selector.js from observables in the consumer's own checkout — hasWebSurface() for `web` (configured navigability routeGlobs, a declared web-framework dependency, or a tracked .html/.css/.jsx/.tsx source file), hasPersistenceLayer() for `data-model` (a declared ORM dependency, a database migrations directory, or a tracked .prisma/.sql schema file), hasPolicyCorpus() for `policies` (configured project.policies.corpusGlobs, or a policy text file under a policies/ or legal/ directory or whose basename carries an explicit policy compound such as privacy-policy or terms-of-service) — and, outside project.policies.corpusGlobs, NOT from an .agentrc key, so a consumer with a real surface keeps these lenses with no configuration. Each probe fails OPEN: an indeterminate answer treats the project as capable, because a wasted lens run is recoverable and silently dropped coverage is not.",
|
|
64
|
+
"enum": ["web", "data-model", "policies", "integrations", "any"],
|
|
65
|
+
"description": "Target-applicability gate: the kind of project this lens can say anything about. `web` — the lens reads a rendered web surface (pages, routes, HTML/CSS, design tokens) and is skipped entirely on a project that has none, however well its keyword or file-pattern triggers match. `data-model` — the lens reads a persistence layer (ORM models, schema migrations, seed data) and is skipped entirely on a project that has none. `policies` — the lens reads a published customer-facing policy corpus (Terms, Privacy, Cookie, refund/cancellation, consent and waiver text) and is skipped entirely on a project that has none. `integrations` — the lens reads a project's external dependencies (third-party SDKs, outbound calls, platform bindings) and is skipped entirely on a project that has none. `any` — always applicable. ABSENT MEANS `any`: omitting the key never changes a lens's behaviour. Applicability is decided by a probe in lib/audit-suite/selector.js from observables in the consumer's own checkout — hasWebSurface() for `web` (configured navigability routeGlobs, a declared web-framework dependency, or a tracked .html/.css/.jsx/.tsx source file), hasPersistenceLayer() for `data-model` (a declared ORM dependency, a database migrations directory, or a tracked .prisma/.sql schema file), hasPolicyCorpus() for `policies` (configured project.policies.corpusGlobs, or a policy text file under a policies/ or legal/ directory or whose basename carries an explicit policy compound such as privacy-policy or terms-of-service), hasExternalDependencies() for `integrations` (a declared integration SDK or HTTP-client dependency, a platform-binding config such as wrangler.toml or serverless.yml, or an outbound call in non-test source) — and, outside project.policies.corpusGlobs, NOT from an .agentrc key, so a consumer with a real surface keeps these lenses with no configuration. Each probe fails OPEN: an indeterminate answer treats the project as capable, because a wasted lens run is recoverable and silently dropped coverage is not.",
|
|
66
66
|
"default": "any"
|
|
67
67
|
},
|
|
68
68
|
"substitutionKeys": {
|
|
@@ -445,6 +445,80 @@ export function hasPolicyCorpus({ config, projectRoot = PROJECT_ROOT } = {}) {
|
|
|
445
445
|
return result;
|
|
446
446
|
}
|
|
447
447
|
|
|
448
|
+
/** Integration families, matched segment-wise like {@link WEB_FRAMEWORK_PACKAGES}. */
|
|
449
|
+
const INTEGRATION_PACKAGES = Object.freeze(
|
|
450
|
+
'stripe clerk auth0 resend sendgrid postmark mailgun nodemailer twilio telnyx aws-sdk google-cloud pinecone openai anthropic-ai sentry axios got node-fetch undici ky'.split(
|
|
451
|
+
' ',
|
|
452
|
+
),
|
|
453
|
+
);
|
|
454
|
+
|
|
455
|
+
const BINDING_CONFIG_FILE_PATTERN =
|
|
456
|
+
/^(wrangler\.(toml|jsonc?)|serverless\.(ya?ml|ts)|Chart\.yaml)$/;
|
|
457
|
+
|
|
458
|
+
const OUTBOUND_SOURCE_PATTERN = /\.(c?js|mjs|tsx?|py|go)$/;
|
|
459
|
+
|
|
460
|
+
const OUTBOUND_CALL_PATTERN =
|
|
461
|
+
/\bfetch\(|\bhttps?\.request\(|\brequests\.(get|post|put|patch|delete)\(|\bhttpx\.|\bhttp\.Get\(/;
|
|
462
|
+
|
|
463
|
+
const OUTBOUND_SNIFF_BYTES = 64 * 1024;
|
|
464
|
+
|
|
465
|
+
/** @type {Map<string, boolean>} */
|
|
466
|
+
const _externalDepsFsCache = new Map();
|
|
467
|
+
|
|
468
|
+
/** Test-only. */
|
|
469
|
+
export function _resetExternalDependenciesCache() {
|
|
470
|
+
_externalDepsFsCache.clear();
|
|
471
|
+
}
|
|
472
|
+
|
|
473
|
+
const TEST_FILE_PATTERN = /\.(test|spec)\./;
|
|
474
|
+
|
|
475
|
+
/** First {@link OUTBOUND_SNIFF_BYTES} of a file, or `''` when unreadable. */
|
|
476
|
+
function readHead(file) {
|
|
477
|
+
try {
|
|
478
|
+
return readFileSync(file, 'utf8').slice(0, OUTBOUND_SNIFF_BYTES);
|
|
479
|
+
} catch {
|
|
480
|
+
return '';
|
|
481
|
+
}
|
|
482
|
+
}
|
|
483
|
+
|
|
484
|
+
function hasOutboundCall(dir, name) {
|
|
485
|
+
return (
|
|
486
|
+
OUTBOUND_SOURCE_PATTERN.test(name) &&
|
|
487
|
+
!TEST_FILE_PATTERN.test(name) &&
|
|
488
|
+
OUTBOUND_CALL_PATTERN.test(readHead(path.join(dir, name)))
|
|
489
|
+
);
|
|
490
|
+
}
|
|
491
|
+
|
|
492
|
+
function isIntegrationMarker(dir, name) {
|
|
493
|
+
return BINDING_CONFIG_FILE_PATTERN.test(name) || hasOutboundCall(dir, name);
|
|
494
|
+
}
|
|
495
|
+
|
|
496
|
+
/** `null` from either signal is indeterminate, not false — fail open. */
|
|
497
|
+
function probeExternalDependencies(root) {
|
|
498
|
+
return (
|
|
499
|
+
declaresDependencyIn(root, INTEGRATION_PACKAGES) !== false ||
|
|
500
|
+
scanNearRoot(root, { isMarkerFile: isIntegrationMarker }) !== false
|
|
501
|
+
);
|
|
502
|
+
}
|
|
503
|
+
|
|
504
|
+
/**
|
|
505
|
+
* Applicability predicate for `target: "integrations"` lenses: an integration
|
|
506
|
+
* SDK or HTTP-client dependency, a platform-binding config, or an outbound call
|
|
507
|
+
* in non-test source. Fails OPEN like {@link hasWebSurface}.
|
|
508
|
+
*
|
|
509
|
+
* @param {{ config?: object|null, projectRoot?: string }} [params]
|
|
510
|
+
* @returns {boolean}
|
|
511
|
+
*/
|
|
512
|
+
export function hasExternalDependencies({ projectRoot = PROJECT_ROOT } = {}) {
|
|
513
|
+
if (!_externalDepsFsCache.has(projectRoot)) {
|
|
514
|
+
_externalDepsFsCache.set(
|
|
515
|
+
projectRoot,
|
|
516
|
+
probeExternalDependencies(projectRoot),
|
|
517
|
+
);
|
|
518
|
+
}
|
|
519
|
+
return _externalDepsFsCache.get(projectRoot);
|
|
520
|
+
}
|
|
521
|
+
|
|
448
522
|
/** Single file × single glob, with the project's matcher semantics (`dot: true`). */
|
|
449
523
|
export function matchesFilePattern(pattern, file) {
|
|
450
524
|
return picomatch(pattern, { dot: true })(file);
|
|
@@ -531,6 +605,7 @@ export function changeSetLacksSiblingTest(changedFiles) {
|
|
|
531
605
|
* @param {typeof hasWebSurface} [params.hasWebSurfaceFn]
|
|
532
606
|
* @param {typeof hasPersistenceLayer} [params.hasPersistenceLayerFn]
|
|
533
607
|
* @param {typeof hasPolicyCorpus} [params.hasPolicyCorpusFn]
|
|
608
|
+
* @param {typeof hasExternalDependencies} [params.hasExternalDependenciesFn]
|
|
534
609
|
*/
|
|
535
610
|
export async function selectAudits({
|
|
536
611
|
ticketId,
|
|
@@ -545,6 +620,7 @@ export async function selectAudits({
|
|
|
545
620
|
hasWebSurfaceFn = hasWebSurface,
|
|
546
621
|
hasPersistenceLayerFn = hasPersistenceLayer,
|
|
547
622
|
hasPolicyCorpusFn = hasPolicyCorpus,
|
|
623
|
+
hasExternalDependenciesFn = hasExternalDependencies,
|
|
548
624
|
}) {
|
|
549
625
|
const config = resolveConfig();
|
|
550
626
|
const timeoutMs = gitTimeoutMsOverride ?? DEFAULT_GIT_TIMEOUT_MS;
|
|
@@ -594,6 +670,7 @@ export async function selectAudits({
|
|
|
594
670
|
hasWebSurfaceFn,
|
|
595
671
|
hasPersistenceLayerFn,
|
|
596
672
|
hasPolicyCorpusFn,
|
|
673
|
+
hasExternalDependenciesFn,
|
|
597
674
|
}),
|
|
598
675
|
});
|
|
599
676
|
|
|
@@ -740,7 +817,7 @@ function matchesAnyKeyword(keywords, content) {
|
|
|
740
817
|
* Lazily memoized per target, so a roster without a `web` lens never pays the
|
|
741
818
|
* filesystem scan.
|
|
742
819
|
*
|
|
743
|
-
* @param {{ config: object, hasWebSurfaceFn: Function, hasPersistenceLayerFn: Function, hasPolicyCorpusFn: Function }} params
|
|
820
|
+
* @param {{ config: object, hasWebSurfaceFn: Function, hasPersistenceLayerFn: Function, hasPolicyCorpusFn: Function, hasExternalDependenciesFn: Function }} params
|
|
744
821
|
* @returns {(target: string) => boolean}
|
|
745
822
|
*/
|
|
746
823
|
function makeTargetApplicabilityProbe({
|
|
@@ -748,11 +825,13 @@ function makeTargetApplicabilityProbe({
|
|
|
748
825
|
hasWebSurfaceFn,
|
|
749
826
|
hasPersistenceLayerFn,
|
|
750
827
|
hasPolicyCorpusFn,
|
|
828
|
+
hasExternalDependenciesFn,
|
|
751
829
|
}) {
|
|
752
830
|
const probes = {
|
|
753
831
|
web: hasWebSurfaceFn,
|
|
754
832
|
'data-model': hasPersistenceLayerFn,
|
|
755
833
|
policies: hasPolicyCorpusFn,
|
|
834
|
+
integrations: hasExternalDependenciesFn,
|
|
756
835
|
};
|
|
757
836
|
const memo = new Map();
|
|
758
837
|
return (target) => {
|
|
@@ -0,0 +1,290 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: >-
|
|
3
|
+
Audit every external dependency, platform binding and cross-service seam:
|
|
4
|
+
whether failure is bounded, whether the system degrades on purpose, what the
|
|
5
|
+
user sees, and whether a human is told and how fast. A deliberately-global
|
|
6
|
+
lens that builds a dependency inventory and follows each failure to a human.
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Resilience Audit
|
|
10
|
+
|
|
11
|
+
You are a Staff Reliability Engineer auditing this project's **dependency
|
|
12
|
+
barriers and integration points**. A per-file grep can ask whether a call has
|
|
13
|
+
a timeout. It cannot ask whether the retry budget still fits the lane that
|
|
14
|
+
drives it, whether the reconcile job's filter covers every state a failure
|
|
15
|
+
leaves behind, or whether the sink a money-path error lands in pages anyone.
|
|
16
|
+
Those are relationships between places, and they are this lens's primary
|
|
17
|
+
finding class.
|
|
18
|
+
|
|
19
|
+
The shared lens machinery lives in
|
|
20
|
+
[`helpers/audit-lens-core.md`](helpers/audit-lens-core.md). Write the report to
|
|
21
|
+
`{{auditResultsDir}}/audit-resilience-results.md`. Dimension values:
|
|
22
|
+
`Inventory | Barrier | Integrity | Signal Path | Landing | Background Lane |
|
|
23
|
+
Assumption Drift`. Every finding adds a **Dependency:** field naming the
|
|
24
|
+
inventory row it cites. The report adds four mandatory sections: **Dependency
|
|
25
|
+
Inventory**, **Signal Path**, **Corrections** and **Verified strengths** (see
|
|
26
|
+
Report additions).
|
|
27
|
+
|
|
28
|
+
## Whole-project scope (global lens)
|
|
29
|
+
|
|
30
|
+
A barrier and the lane or sink it depends on sit in files no change set names,
|
|
31
|
+
so this lens **always evaluates the whole project**. It declares
|
|
32
|
+
`"scope": "global"` and `"target": "integrations"` in
|
|
33
|
+
[`audit-rules.json`](../schemas/audit-rules.json): the selector skips it on a
|
|
34
|
+
project with no integration SDK, no outbound call outside tests and no
|
|
35
|
+
platform-binding config, per the core's absent-vs-inapplicable rule.
|
|
36
|
+
|
|
37
|
+
```text
|
|
38
|
+
{{changedFiles}}
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
- For this lens, **ignore** the `{{changedFiles}}` block above as a filter even
|
|
42
|
+
when it is populated: the project is evaluated whole. When populated, use it
|
|
43
|
+
as a **prioritization hint** — walk the dependencies a changed file touches
|
|
44
|
+
first — never as a limit on what is audited or reported.
|
|
45
|
+
|
|
46
|
+
## Execution strategy
|
|
47
|
+
|
|
48
|
+
Run this lens as one `subagent_type: auditor` dispatch per the core's
|
|
49
|
+
Execution strategy: the auditor builds the inventory (Step 1), then judges the
|
|
50
|
+
dimensions. Fan out only when the operator's invocation asks for it. The
|
|
51
|
+
caller then builds the inventory first and dispatches one auditor per
|
|
52
|
+
dimension, or per dependency family on a large inventory, in a single turn.
|
|
53
|
+
Each child receives the inventory table as context, and the caller merges the
|
|
54
|
+
results under the core's self-cross-check.
|
|
55
|
+
|
|
56
|
+
Two invocation modes narrow a manual run:
|
|
57
|
+
|
|
58
|
+
- `--inventory-only` runs Step 1 alone and writes only the Dependency
|
|
59
|
+
Inventory: no findings and no judgement. It is a cheap standing artifact a
|
|
60
|
+
consumer can diff between sweeps.
|
|
61
|
+
- `--dependency <name>` narrows the inventory and every dimension to the one
|
|
62
|
+
named dependency.
|
|
63
|
+
|
|
64
|
+
## Step 1: Dependency inventory (tool-first)
|
|
65
|
+
|
|
66
|
+
Build the inventory before judging anything. Every later finding cites a row.
|
|
67
|
+
|
|
68
|
+
- Census the dependencies:
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
# Third-party SDK imports (JS/TS; adapt the pattern for Python/Go)
|
|
72
|
+
rg -n --glob '!**/*.{test,spec}.*' "from ['\"](@?[a-z0-9-]+)(/[^'\"]*)?['\"]" | \
|
|
73
|
+
rg -i "stripe|clerk|auth0|resend|sendgrid|postmark|mailgun|twilio|telnyx|aws-sdk|google-cloud|pinecone|openai|anthropic|sentry|libsql|turso"
|
|
74
|
+
# Outbound call sites outside tests
|
|
75
|
+
rg -n --glob '!**/*.{test,spec}.*' --glob '!**/{test,tests,__tests__}/**' \
|
|
76
|
+
"\b(fetch|axios(\.\w+)?|got|ky|http\.request|https\.request)\s*\(|requests\.(get|post)|httpx\.|http\.Get\("
|
|
77
|
+
# Bindings and secrets the runtime expects
|
|
78
|
+
ls wrangler.* serverless.* .env.example 2>/dev/null; rg -n "^\[\[?(kv_namespaces|r2_buckets|d1_databases|vectorize|ai|queues|services)" --glob 'wrangler.*'
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Emit one row per dependency in the **Dependency Inventory** table (columns
|
|
82
|
+
under Report additions): its callers, its timeout and whether that timeout
|
|
83
|
+
covers the body read, its retry posture, what the system does when it fails,
|
|
84
|
+
what it does when its secret or binding is absent, the user-visible message,
|
|
85
|
+
the alert path, and the detection lag for a full outage.
|
|
86
|
+
|
|
87
|
+
## Step 2: Evaluation dimensions
|
|
88
|
+
|
|
89
|
+
Every finding takes exactly one of these as its `Dimension`. Inventory is the
|
|
90
|
+
artifact itself; a missing or contradictory row (a caller with no timeout
|
|
91
|
+
recorded anywhere) is the only Inventory finding.
|
|
92
|
+
|
|
93
|
+
### Barrier
|
|
94
|
+
|
|
95
|
+
- Run the barrier greps:
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
# Outbound calls with no timeout or signal on the same line
|
|
99
|
+
rg -n "\b(fetch|axios(\.\w+)?|http\.request|got|ky)\s*\(" --glob '!**/*.test.*' | \
|
|
100
|
+
rg -v -i "timeout|signal|AbortController|AbortSignal"
|
|
101
|
+
# Empty catch blocks (a failure that leaves no signal)
|
|
102
|
+
rg -n "catch\s*(\([^)]*\))?\s*\{\s*\}"
|
|
103
|
+
# Retry and backoff sites (check each is bounded and idempotent)
|
|
104
|
+
rg -n -i "retr(y|ies)|backoff|circuit.?breaker|p-retry"
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
- **Timeout scope** — a timeout that covers the headers but not the body read
|
|
108
|
+
is a partial barrier; a slow body still holds the request.
|
|
109
|
+
- **Retry safety** — a retry that can replay an ambiguous write without an
|
|
110
|
+
idempotency key.
|
|
111
|
+
- **Outage as empty** — a deadline or failure path that resolves to an empty
|
|
112
|
+
result instead of rejecting. An outage must never read as "no rows".
|
|
113
|
+
- **Fail-open vs fail-closed** — left implicit at a seam where the choice
|
|
114
|
+
matters.
|
|
115
|
+
- **Missing secret or binding** — crashes, or silently disables, instead of
|
|
116
|
+
entering an explicit disabled mode with a reason.
|
|
117
|
+
- **Bypassed primitive** — a raw call that skips the project's own bounded
|
|
118
|
+
fetch helper when one exists.
|
|
119
|
+
|
|
120
|
+
### Integrity
|
|
121
|
+
|
|
122
|
+
Money and safety under partial failure, wherever an external side effect is
|
|
123
|
+
involved:
|
|
124
|
+
|
|
125
|
+
- **Idempotency keys** on every money-moving write.
|
|
126
|
+
- **Claim before transmit** on every external send (email, SMS, push), so a
|
|
127
|
+
crash between transmit and ledger cannot duplicate.
|
|
128
|
+
- **Webhook receivers** return a 5xx on downstream failure so the provider
|
|
129
|
+
retries, and keep a replay ledger. A 200 that loses the event is the
|
|
130
|
+
finding.
|
|
131
|
+
- **Reconciliation coverage** — the job exists **and its candidate set covers
|
|
132
|
+
every state a failure can leave**, not only the happy-path pending state.
|
|
133
|
+
- **Compensation** — a multi-step write cleans up the blob store when the
|
|
134
|
+
record write fails.
|
|
135
|
+
- **Fan-out settlement** — per recipient, not "any recipient was reached".
|
|
136
|
+
|
|
137
|
+
### Signal Path
|
|
138
|
+
|
|
139
|
+
For each failure class in the inventory, trace signal → sink → rule → human
|
|
140
|
+
and name the lag.
|
|
141
|
+
|
|
142
|
+
- **Sink classification** — a log store with bounded retention and no
|
|
143
|
+
alerting is **not** an alert, whatever level the line carries.
|
|
144
|
+
- **Capture census** — every error-tracker capture site by level and
|
|
145
|
+
fingerprint: money or safety conditions captured at `warning`, captured with
|
|
146
|
+
an opt-out tag, or not captured at all.
|
|
147
|
+
- **Storms and collapse** — un-fingerprinted per-request captures that storm,
|
|
148
|
+
and shared helpers whose `new Error(name)` collapses distinct failures into
|
|
149
|
+
one group.
|
|
150
|
+
- **Silent producers** — alert queries that swallow their own failure
|
|
151
|
+
(`catch { return 0 }`), so a broken alert reads as healthy.
|
|
152
|
+
- **Sink posture** — the sink's own storm guard and quota on its tier; one
|
|
153
|
+
flood must not blind every other pager.
|
|
154
|
+
- **Watch the watcher** — walk every hop of the alert pipeline (tracker →
|
|
155
|
+
relay → issue tracker → chat) and ask what fires when that hop fails. Find
|
|
156
|
+
first-seen-only rules that miss regressions.
|
|
157
|
+
|
|
158
|
+
### Landing
|
|
159
|
+
|
|
160
|
+
- **Error page** — every app ships a branded error page; an SSR throw is never
|
|
161
|
+
a blank body.
|
|
162
|
+
- **Outage shape** — an outage never borrows the empty-state shape ("you have
|
|
163
|
+
nothing" on a database failure).
|
|
164
|
+
- **Retry affordance** — the client honours a server-sent `503` with
|
|
165
|
+
`Retry-After`, and a transient failure offers a retry.
|
|
166
|
+
- **Error boundaries** on revenue and safety flows.
|
|
167
|
+
- **Report path** — "report a problem" still works when the primary store is
|
|
168
|
+
down, and the copy names what to do.
|
|
169
|
+
|
|
170
|
+
### Background Lane
|
|
171
|
+
|
|
172
|
+
- **Per-job isolation** — one throw does not stop the other jobs in a fire.
|
|
173
|
+
- **Per-job deadline** — one hung upstream cannot consume the whole fire.
|
|
174
|
+
- **Overlap safety** and at-most-once claims around side effects.
|
|
175
|
+
- **Heartbeats** with explicit completed-vs-passed semantics, and a streak
|
|
176
|
+
threshold so a job failing every fire for days is noticed rather than
|
|
177
|
+
grouped into one issue created on day one.
|
|
178
|
+
- **Missing secret** skips the job with a reason; it never crashes the fire.
|
|
179
|
+
|
|
180
|
+
### Assumption Drift
|
|
181
|
+
|
|
182
|
+
Constants and comments justified by a fact the tree has since changed: a retry
|
|
183
|
+
budget "chosen against the hourly cadence" after the drain moved to an
|
|
184
|
+
every-minute lane, "same trade as the sibling app" after the sibling was
|
|
185
|
+
fixed, a grace window tuned to a schedule that was later shortened. No other
|
|
186
|
+
lens owns this pass, and it is cheap:
|
|
187
|
+
|
|
188
|
+
- Find the drift candidates:
|
|
189
|
+
|
|
190
|
+
```bash
|
|
191
|
+
# Cadence and parity words next to constants or config
|
|
192
|
+
rg -n -i -B2 -A2 "hourly|every (minute|hour|day)|\bcron\b|cadence|same as|mirrors|like the sibling" \
|
|
193
|
+
--glob '!**/*.{test,spec}.*' --glob '!**/*.md' | rg -i "[0-9]|const |= "
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
For every hit, find the fact the comment cites (the cron expression, the lane
|
|
197
|
+
membership, the sibling's current code) and check it still holds. A constant
|
|
198
|
+
whose justification no longer holds is the finding; cite both places.
|
|
199
|
+
|
|
200
|
+
## Step 3: Evidence and severity
|
|
201
|
+
|
|
202
|
+
This lens adds four rules to the core's self-cross-check:
|
|
203
|
+
|
|
204
|
+
1. **Second read for every Critical and High.** Re-verify each one by reading
|
|
205
|
+
the actual control-flow order (middleware sequence, call order, lane
|
|
206
|
+
membership), not a comment or a single file. A boundary that looks like it
|
|
207
|
+
swallows a throw may sit outside the wrapper that captures it. A Critical
|
|
208
|
+
or High that fails the second read is dropped and recorded under
|
|
209
|
+
`## Corrections` with the reason, so the next sweep does not re-raise it.
|
|
210
|
+
2. **Alert gaps name the route.** An alert-gap finding names the sink, its
|
|
211
|
+
retention or level, and the rule (or the absence of one) that would carry it
|
|
212
|
+
to a human. "Not in the error tracker" alone is not a finding.
|
|
213
|
+
3. **Timeouts name the cost.** A missing-timeout finding names what the hang
|
|
214
|
+
holds: a request, a claim, a cron fire. An unbounded call with no downstream
|
|
215
|
+
consequence is `Low`.
|
|
216
|
+
4. **Severity calibration**:
|
|
217
|
+
`Critical` or `High` — money or safety integrity, a whole-surface outage, or
|
|
218
|
+
a missing human signal for either. `Medium` — degraded UX with no feedback,
|
|
219
|
+
or a signal that arrives hours late. `Low` — polish and consistency.
|
|
220
|
+
|
|
221
|
+
## Step 4: Boundaries with sibling lenses
|
|
222
|
+
|
|
223
|
+
Drop anything a sibling owns whole, and cite that sibling when you do:
|
|
224
|
+
|
|
225
|
+
- `audit-sre` keeps rollback, migration reversibility, kill switches,
|
|
226
|
+
runbooks, health endpoints, graceful shutdown and the release-readiness
|
|
227
|
+
checklist.
|
|
228
|
+
- `audit-performance` keeps TOCTOU, unawaited promises and non-atomic writes;
|
|
229
|
+
this lens asks the integrity question only where an external side effect is
|
|
230
|
+
involved.
|
|
231
|
+
- `audit-security` keeps webhook signature verification; this lens asks about
|
|
232
|
+
the response code on downstream failure, the replay ledger and the retry
|
|
233
|
+
contract, not the crypto.
|
|
234
|
+
- `audit-privacy` keeps what the sinks contain; this lens asks where they go
|
|
235
|
+
and whether anyone reads them.
|
|
236
|
+
- `audit-devops` keeps CI and deploy pipeline resilience.
|
|
237
|
+
|
|
238
|
+
## Constraint
|
|
239
|
+
|
|
240
|
+
Read-only and static: no live probing of vendors, no remediation. A consumer's
|
|
241
|
+
smoke lane is a finding input, not something this lens runs.
|
|
242
|
+
|
|
243
|
+
## Report additions
|
|
244
|
+
|
|
245
|
+
Beyond the core's shared skeleton, this lens adds its title, the four
|
|
246
|
+
mandatory sections and the `Dependency` finding field. Write `_No findings._`
|
|
247
|
+
under a section rather than omitting it:
|
|
248
|
+
|
|
249
|
+
```markdown
|
|
250
|
+
# Resilience Audit Report
|
|
251
|
+
|
|
252
|
+
## Executive Summary
|
|
253
|
+
|
|
254
|
+
[Headline read.]
|
|
255
|
+
|
|
256
|
+
Self-cross-check: kept <k> / dropped <d>.
|
|
257
|
+
Severity tally: Critical <n> / High <n> / Medium <n> / Low <n>
|
|
258
|
+
|
|
259
|
+
## Dependency Inventory
|
|
260
|
+
|
|
261
|
+
| Dependency | Callers | Timeout (covers body?) | Retry (bounded / backoff / idempotent) | On failure | On missing secret | User sees | Alert path | Detection lag |
|
|
262
|
+
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
|
263
|
+
| [name] | `path:line` | [value — yes/no] | [posture] | [degraded mode] | [disabled mode] | [message] | [sink → rule → human] | [lag] |
|
|
264
|
+
|
|
265
|
+
## Signal Path
|
|
266
|
+
|
|
267
|
+
| Failure class | Signal | Reaches a human via | Detection lag | Gap |
|
|
268
|
+
| --- | --- | --- | --- | --- |
|
|
269
|
+
| [class] | [capture / log / metric] | [sink → rule → channel] | [lag] | [gap or none] |
|
|
270
|
+
|
|
271
|
+
## Detailed Findings
|
|
272
|
+
|
|
273
|
+
### `path/to/primary-file.ext` — [Short title of the issue]
|
|
274
|
+
|
|
275
|
+
- **Dimension:** [Inventory | Barrier | Integrity | Signal Path | Landing | Background Lane | Assumption Drift]
|
|
276
|
+
- **Severity:** [Critical | High | Medium | Low | Info]
|
|
277
|
+
- **Dependency:** [the inventory row this finding cites]
|
|
278
|
+
- [Location, Current State, Recommendation & Rationale, Acceptance signal —
|
|
279
|
+
per the core skeleton]
|
|
280
|
+
|
|
281
|
+
## Corrections
|
|
282
|
+
|
|
283
|
+
| Dropped finding | Severity as drafted | Why the second read cleared it |
|
|
284
|
+
| --- | --- | --- |
|
|
285
|
+
| [title] | [High] | [the control-flow order actually read, with `path:line`] |
|
|
286
|
+
|
|
287
|
+
## Verified strengths
|
|
288
|
+
|
|
289
|
+
- [A primitive that works and that fix PRs must not erode — `path:line`]
|
|
290
|
+
```
|
|
@@ -6,8 +6,7 @@ description: "Audit production-readiness for a release candidate: SLOs, observab
|
|
|
6
6
|
|
|
7
7
|
You are a Senior SRE & Lead Developer running an operational-readiness audit for
|
|
8
8
|
a production release candidate — rollback & recovery paths, observability &
|
|
9
|
-
instrumentation,
|
|
10
|
-
docs. The shared lens machinery lives in
|
|
9
|
+
instrumentation, graceful shutdown, and runbooks & operational docs. The shared lens machinery lives in
|
|
11
10
|
[`helpers/audit-lens-core.md`](helpers/audit-lens-core.md). Write the report to
|
|
12
11
|
`{{auditResultsDir}}/audit-sre-results.md`. Each finding carries a **Category:**
|
|
13
12
|
(`Rollback & Recovery | Observability | Resilience | Runbooks`); the report adds
|
|
@@ -27,27 +26,22 @@ Per the core's Scope interpretation:
|
|
|
27
26
|
Run this lens as one `subagent_type: auditor` dispatch per the core's
|
|
28
27
|
Execution strategy.
|
|
29
28
|
|
|
30
|
-
## Step 1:
|
|
29
|
+
## Step 1: Operational Readiness Battery (Read-Only, Tool-First)
|
|
31
30
|
|
|
32
31
|
This lens audits **operational readiness** — can this release be observed,
|
|
33
32
|
survive failure, and be rolled back? It deliberately does **not** re-audit
|
|
34
33
|
secrets, injection, dead code, or complexity; those are owned by
|
|
35
|
-
[`audit-security`](audit-security.md) and [`audit-clean-code`](audit-clean-code.md)
|
|
36
|
-
|
|
34
|
+
[`audit-security`](audit-security.md) and [`audit-clean-code`](audit-clean-code.md),
|
|
35
|
+
and it leaves dependency barriers (timeouts, retries, swallowed failures) and
|
|
36
|
+
the signal path from a failure to a human to
|
|
37
|
+
[`audit-resilience`](audit-resilience.md). Ground findings in the greps below, then read the operational surfaces they
|
|
37
38
|
flag.
|
|
38
39
|
|
|
39
|
-
1. **Run the
|
|
40
|
+
1. **Run the readiness battery.** Each grep maps to a Step 2 dimension:
|
|
40
41
|
|
|
41
42
|
```bash
|
|
42
|
-
# Network calls without a timeout (a hung upstream stalls the whole request)
|
|
43
|
-
rg -n "\b(fetch|axios(\.\w+)?|http\.request|got|ky)\s*\(" --glob '!**/*.test.*' | \
|
|
44
|
-
rg -v -i "timeout|signal|AbortController"
|
|
45
43
|
# Graceful-shutdown handlers (their ABSENCE is the finding for long-lived processes)
|
|
46
44
|
rg -n "process\.on\(\s*['\"]SIG(TERM|INT)['\"]" || echo "no SIGTERM/SIGINT handler found"
|
|
47
|
-
# Error-swallowing empty catch blocks (silent failure — no signal to observe)
|
|
48
|
-
rg -n "catch\s*(\([^)]*\))?\s*\{\s*\}"
|
|
49
|
-
# Retry / backoff on network-dependent work (resilience to transient failure)
|
|
50
|
-
rg -n -i "retr(y|ies)|backoff|circuit.?breaker|p-retry"
|
|
51
45
|
# Health / readiness endpoints (needed for orchestrated rollout & rollback)
|
|
52
46
|
rg -n -i "/(health|healthz|readyz|livez|ping)\b|healthCheck"
|
|
53
47
|
```
|
|
@@ -80,22 +74,19 @@ Evaluate the release candidate against these **production-readiness** criteria.
|
|
|
80
74
|
|
|
81
75
|
### 3. Resilience & Failure Handling
|
|
82
76
|
|
|
83
|
-
- **Timeouts & Cancellation:** Every outbound call needs a timeout /
|
|
84
|
-
`AbortSignal` (grep 1 in Step 1). A call without one is a Reliability finding.
|
|
85
|
-
- **Retry & Backoff:** Transient-failure-prone calls should retry with backoff;
|
|
86
|
-
flag network work that fails hard on the first error.
|
|
87
77
|
- **Graceful Shutdown:** Long-lived processes must handle `SIGTERM`/`SIGINT`
|
|
88
|
-
and drain in-flight work (grep
|
|
78
|
+
and drain in-flight work (grep 1). Its absence risks dropped requests on
|
|
89
79
|
every deploy.
|
|
90
|
-
- **
|
|
91
|
-
|
|
80
|
+
- **Dependency barriers** belong to `audit-resilience`: timeouts, retries,
|
|
81
|
+
swallowed failures, integrity under partial failure and the alert path. Drop
|
|
82
|
+
them here.
|
|
92
83
|
|
|
93
84
|
### 4. Runbooks & Operational Docs
|
|
94
85
|
|
|
95
86
|
- **Runbook Coverage:** Does an operational runbook exist for this
|
|
96
87
|
service/feature (how to deploy, roll back, and respond to the top failure
|
|
97
88
|
modes)? A new production surface with no runbook is a finding.
|
|
98
|
-
- **Health & Readiness:** Are health/readiness endpoints (grep
|
|
89
|
+
- **Health & Readiness:** Are health/readiness endpoints (grep 2) present and
|
|
99
90
|
wired into the orchestrator so a bad rollout is caught before it takes
|
|
100
91
|
traffic?
|
|
101
92
|
- **On-Call Escalation:** Is ownership / escalation for this surface documented?
|
package/docs/CHANGELOG.md
CHANGED
|
@@ -15,6 +15,13 @@ All notable changes to this project will be documented in this file.
|
|
|
15
15
|
-->
|
|
16
16
|
<!-- markdownlint-disable-file MD004 MD012 MD037 -->
|
|
17
17
|
|
|
18
|
+
## [2.102.0](https://github.com/dsj1984/mandrel/compare/mandrel-v2.101.0...mandrel-v2.102.0) (2026-10-23)
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
### Added
|
|
22
|
+
|
|
23
|
+
* add /audit-resilience: a global lens that inventories every external dependency and follows each failure through its barrier to a human ([#5892](https://github.com/dsj1984/mandrel/issues/5892)) ([#5893](https://github.com/dsj1984/mandrel/issues/5893)) ([7a52cfc](https://github.com/dsj1984/mandrel/commit/7a52cfc123670a2b91d08c6310c9d26c8bc25988))
|
|
24
|
+
|
|
18
25
|
## [2.101.0](https://github.com/dsj1984/mandrel/compare/mandrel-v2.100.0...mandrel-v2.101.0) (2026-10-23)
|
|
19
26
|
|
|
20
27
|
|
package/package.json
CHANGED