ronce-racine 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +90 -0
- package/LICENSE +21 -0
- package/README.md +281 -0
- package/agents/code-reviewer.md +33 -0
- package/agents/qa-tester.md +28 -0
- package/dist/hooks/README.md +182 -0
- package/dist/hooks/bash-npm-silent.mjs +81 -0
- package/dist/hooks/session-inject.mjs +53 -0
- package/dist/hooks/session-precompact.mjs +36 -0
- package/dist/hooks/session-writer.mjs +142 -0
- package/dist/hooks/skill-reminder.mjs +89 -0
- package/dist/hooks/truncate-bash-output.mjs +46 -0
- package/dist/hooks/truncate-output.mjs +88 -0
- package/dist/hooks/worktree-env-setup.mjs +123 -0
- package/dist/install.js +962 -0
- package/docs/adopting-a-repo.md +80 -0
- package/docs/architecture.md +76 -0
- package/docs/decisions.md +72 -0
- package/docs/developing.md +143 -0
- package/docs/evaluating-skills.md +41 -0
- package/docs/git-worktrees.md +62 -0
- package/docs/rules/clean-architecture-deps.md +66 -0
- package/docs/rules/commits.md +59 -0
- package/docs/rules/detection-gap-protocol.md +55 -0
- package/docs/rules/error-handling-discipline.md +62 -0
- package/docs/rules/minimal-code.md +58 -0
- package/docs/rules/no-raw-sql-interpolation.md +57 -0
- package/docs/rules/pre-commit-secret-detection.md +54 -0
- package/docs/rules/secure-logging.md +53 -0
- package/docs/rules/sql-migrations-discipline.md +58 -0
- package/docs/rules/subscription-cleanup.md +63 -0
- package/docs/rules/test-discipline.md +66 -0
- package/docs/rules/ui-states-complete.md +52 -0
- package/docs/templates/SKILL.template.md +66 -0
- package/docs/templates/rule.template.md +16 -0
- package/docs/writing-a-rule.md +43 -0
- package/docs/writing-a-skill.md +118 -0
- package/package.json +45 -0
- package/rules/clean-architecture-deps.md +20 -0
- package/rules/commits.md +16 -0
- package/rules/detection-gap-protocol.md +35 -0
- package/rules/error-handling-discipline.md +23 -0
- package/rules/minimal-code.md +32 -0
- package/rules/no-raw-sql-interpolation.md +20 -0
- package/rules/pre-commit-secret-detection.md +14 -0
- package/rules/secure-logging.md +16 -0
- package/rules/sql-migrations-discipline.md +18 -0
- package/rules/subscription-cleanup.md +22 -0
- package/rules/test-discipline.md +20 -0
- package/rules/ui-states-complete.md +23 -0
- package/scripts/subscription-leak-scan.ts +53 -0
- package/skills/adversarial-feature-challenge/README.md +65 -0
- package/skills/adversarial-feature-challenge/SKILL.md +96 -0
- package/skills/adversarial-feature-challenge/assets/fuzz-payloads.txt +60 -0
- package/skills/adversarial-feature-challenge/templates/challenge-report.md +42 -0
- package/skills/api-contract-versioning/README.md +62 -0
- package/skills/api-contract-versioning/SKILL.md +76 -0
- package/skills/api-contract-versioning/templates/consumer-inventory.md +29 -0
- package/skills/audit-architecture/README.md +56 -0
- package/skills/audit-architecture/SKILL.md +97 -0
- package/skills/audit-architecture/reference/ar-architecture.md +275 -0
- package/skills/audit-architecture/reference/di-disponibilite.md +252 -0
- package/skills/audit-architecture/reference/im-incidents.md +217 -0
- package/skills/audit-architecture/reference/pa-pannes.md +362 -0
- package/skills/audit-architecture/reference/re-recuperation.md +300 -0
- package/skills/audit-architecture/reference/sc-scalabilite.md +173 -0
- package/skills/audit-architecture/reference/su-support.md +180 -0
- package/skills/audit-ci-cd/README.md +54 -0
- package/skills/audit-ci-cd/SKILL.md +124 -0
- package/skills/audit-ci-cd/reference/artefacts.md +39 -0
- package/skills/audit-ci-cd/reference/deploiement.md +100 -0
- package/skills/audit-ci-cd/reference/dora.md +94 -0
- package/skills/audit-ci-cd/reference/iac.md +40 -0
- package/skills/audit-ci-cd/reference/pipeline-ci.md +36 -0
- package/skills/audit-ci-cd/reference/supply-chain.md +44 -0
- package/skills/audit-compliance/README.md +69 -0
- package/skills/audit-compliance/SKILL.md +176 -0
- package/skills/audit-compliance/reference/conformite-rgpd.md +295 -0
- package/skills/audit-compliance/reference/finops.md +318 -0
- package/skills/audit-compliance/reference/gouvernance-donnees.md +238 -0
- package/skills/audit-industrialisation/README.md +53 -0
- package/skills/audit-industrialisation/SKILL.md +251 -0
- package/skills/audit-observability/README.md +64 -0
- package/skills/audit-observability/SKILL.md +134 -0
- package/skills/audit-observability/reference/alerting.md +214 -0
- package/skills/audit-observability/reference/logging.md +114 -0
- package/skills/audit-observability/reference/metriques.md +166 -0
- package/skills/audit-observability/reference/traces-dashboards.md +90 -0
- package/skills/audit-performance-frontend/README.md +91 -0
- package/skills/audit-performance-frontend/SKILL.md +194 -0
- package/skills/audit-performance-frontend/reference/assets-cache-network.md +129 -0
- package/skills/audit-performance-frontend/reference/css-rendering-pipeline.md +154 -0
- package/skills/audit-performance-frontend/reference/html-critical-path.md +128 -0
- package/skills/audit-performance-frontend/reference/js-engine-runtime.md +280 -0
- package/skills/audit-performance-frontend/reference/memory-gc.md +126 -0
- package/skills/audit-performance-frontend/reference/profiling-tooling.md +98 -0
- package/skills/audit-quality/README.md +85 -0
- package/skills/audit-quality/SKILL.md +126 -0
- package/skills/audit-quality/reference/qa-devops.md +539 -0
- package/skills/audit-quality/reference/qu-qualite-donnees.md +402 -0
- package/skills/audit-report/README.md +72 -0
- package/skills/audit-report/SKILL.md +241 -0
- package/skills/audit-report/reference/scoring-model.md +45 -0
- package/skills/audit-security/README.md +72 -0
- package/skills/audit-security/SKILL.md +140 -0
- package/skills/audit-security/reference/auth-identite.md +98 -0
- package/skills/audit-security/reference/autorisation.md +66 -0
- package/skills/audit-security/reference/chiffrement-tls.md +97 -0
- package/skills/audit-security/reference/detection-patching.md +67 -0
- package/skills/audit-security/reference/incident-response.md +98 -0
- package/skills/audit-security/reference/reseau-conteneurs.md +72 -0
- package/skills/audit-security/reference/secrets.md +36 -0
- package/skills/audit-testing/README.md +82 -0
- package/skills/audit-testing/SKILL.md +131 -0
- package/skills/audit-testing/reference/golden-dataset.md +65 -0
- package/skills/audit-testing/reference/resilience-chaos.md +93 -0
- package/skills/audit-testing/reference/tests-contractuels.md +65 -0
- package/skills/audit-testing/reference/tests-e2e-flaky.md +67 -0
- package/skills/audit-testing/reference/tests-unit-integ-ci.md +125 -0
- package/skills/bug-ticket-root-cause/README.md +67 -0
- package/skills/bug-ticket-root-cause/SKILL.md +86 -0
- package/skills/bug-ticket-root-cause/templates/bug-ticket.md +43 -0
- package/skills/bug-triage-structured/README.md +63 -0
- package/skills/bug-triage-structured/SKILL.md +104 -0
- package/skills/bug-triage-structured/templates/bug-triage.md +38 -0
- package/skills/ci-pipeline-orchestration/README.md +55 -0
- package/skills/ci-pipeline-orchestration/SKILL.md +110 -0
- package/skills/commit-readiness-review/README.md +66 -0
- package/skills/commit-readiness-review/SKILL.md +82 -0
- package/skills/commit-readiness-review/scripts/precommit-scan.ts +81 -0
- package/skills/comprehensive-test-strategy/README.md +64 -0
- package/skills/comprehensive-test-strategy/SKILL.md +108 -0
- package/skills/comprehensive-test-strategy/reference/testing-advanced.md +85 -0
- package/skills/daily-workflow-optimization/README.md +56 -0
- package/skills/daily-workflow-optimization/SKILL.md +80 -0
- package/skills/database-schema-evolution/README.md +54 -0
- package/skills/database-schema-evolution/SKILL.md +107 -0
- package/skills/ddd-backend-implementation/README.md +63 -0
- package/skills/ddd-backend-implementation/SKILL.md +95 -0
- package/skills/design-system-component-lifecycle/README.md +56 -0
- package/skills/design-system-component-lifecycle/SKILL.md +97 -0
- package/skills/design-system-component-lifecycle/templates/component-contract.md +43 -0
- package/skills/detection-sweep/README.md +58 -0
- package/skills/detection-sweep/SKILL.md +93 -0
- package/skills/detection-sweep/scripts/sweep.ts +80 -0
- package/skills/domain-modeling-design/README.md +57 -0
- package/skills/domain-modeling-design/SKILL.md +83 -0
- package/skills/frontend-fullstack-implementation/README.md +60 -0
- package/skills/frontend-fullstack-implementation/SKILL.md +133 -0
- package/skills/frontend-spec-call-site-audit/README.md +66 -0
- package/skills/frontend-spec-call-site-audit/SKILL.md +82 -0
- package/skills/frontend-spec-call-site-audit/scripts/audit-entry-points.ts +79 -0
- package/skills/frontend-spec-call-site-audit/templates/ticket-frontend.md +32 -0
- package/skills/merge-request-review/README.md +60 -0
- package/skills/merge-request-review/SKILL.md +77 -0
- package/skills/merge-request-review/templates/mr-review-report.md +29 -0
- package/skills/production-incident-diagnostic/README.md +63 -0
- package/skills/production-incident-diagnostic/SKILL.md +103 -0
- package/skills/production-incident-diagnostic/templates/incident-report.md +33 -0
- package/skills/qa-session-intake/README.md +56 -0
- package/skills/qa-session-intake/SKILL.md +78 -0
- package/skills/recording-decisions/README.md +91 -0
- package/skills/recording-decisions/SKILL.md +84 -0
- package/skills/recurring-bug-root-cause/README.md +64 -0
- package/skills/recurring-bug-root-cause/SKILL.md +93 -0
- package/skills/recurring-bug-root-cause/scripts/detect-recurring-fixes.ts +46 -0
- package/skills/recurring-bug-root-cause/templates/postmortem-recidive.md +37 -0
- package/skills/refactoring-shared-component-api/README.md +50 -0
- package/skills/refactoring-shared-component-api/SKILL.md +84 -0
- package/skills/refactoring-shared-component-api/scripts/sweep-call-sites.ts +85 -0
- package/skills/refactoring-shared-component-api/templates/migration-checklist.md +19 -0
- package/skills/validating-features-end-to-end/README.md +50 -0
- package/skills/validating-features-end-to-end/SKILL.md +90 -0
- package/skills/visual-regression-check/README.md +51 -0
- package/skills/visual-regression-check/SKILL.md +89 -0
- package/skills/writing-robust-tests/README.md +53 -0
- package/skills/writing-robust-tests/SKILL.md +88 -0
- package/templates/.adopted.example +8 -0
- package/templates/anti-drift.gitlab-ci.yml +23 -0
|
@@ -0,0 +1,251 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: audit-industrialisation
|
|
3
|
+
description: Full software-industrialisation audit. Orchestrates the 8 domain audits and produces a consolidated report. Use when running a full audit of a project's software engineering maturity. Triggers on "audit industrialisation", "industrialisation audit", "audit complet", "full audit", "audit projet", "project audit", "audit maturité", "maturity audit", "audit global", "global audit".
|
|
4
|
+
version: 1.0.1
|
|
5
|
+
metadata:
|
|
6
|
+
last-reviewed: 2026-07-20
|
|
7
|
+
category: audit
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# Software industrialisation audit
|
|
11
|
+
|
|
12
|
+
Full audit of a software project's industrialisation maturity. Orchestrates the 8 domain audits through parallel subagents and produces a consolidated report.
|
|
13
|
+
|
|
14
|
+
**214 questions, 18 sections, 8 domains, 0-4 maturity model.**
|
|
15
|
+
|
|
16
|
+
## This skill vs. a domain audit
|
|
17
|
+
|
|
18
|
+
- **This skill** when: a global, multi-domain audit - it profiles the project, orchestrates the 8 domain audits in parallel and consolidates the report
|
|
19
|
+
- **A single `audit-<domain>`** instead if: the scope targets one domain (e.g. just `audit-security`)
|
|
20
|
+
- **`audit-report`**: NOT invoked - Phase 3 reuses its template and its `audit-report/reference/scoring-model.md` directly (see Phase 3)
|
|
21
|
+
|
|
22
|
+
## Where the questions come from
|
|
23
|
+
|
|
24
|
+
The 214 questions are **defined in the audit skills themselves**: each `audit-<domain>` and its `reference/<section>.md` files carry the wording, the criticality and the 0-4 levels, with stable IDs (`ci-01`, `te-09a`, `PF-21`… - casing varies by domain grid; compare IDs case-insensitively). This orchestrator **defines no questions** - it profiles the project, launches the audits and consolidates. The IDs are identical across the orchestrator, the skills and the report.
|
|
25
|
+
|
|
26
|
+
### Coverage per skill
|
|
27
|
+
|
|
28
|
+
| Skill | Sections | Question count |
|
|
29
|
+
|-------|--------|-------------|
|
|
30
|
+
| `audit-ci-cd` | ci | 10 |
|
|
31
|
+
| `audit-testing` | te | 13 |
|
|
32
|
+
| `audit-security` | se | 16 |
|
|
33
|
+
| `audit-observability` | ob, al | 15 |
|
|
34
|
+
| `audit-architecture` | ar, sc, pa, re, di, su, im | 74 |
|
|
35
|
+
| `audit-quality` | qa, qu | 28 |
|
|
36
|
+
| `audit-compliance` | co, dg, fi | 29 |
|
|
37
|
+
| `audit-performance-frontend` | pf | 29 |
|
|
38
|
+
|
|
39
|
+
**Total: 214 questions across 18 sections, 8 domains.**
|
|
40
|
+
|
|
41
|
+
> A project that manages its audits in a dedicated tool (external knowledge base, audit app) adds the export to that tool in its **own project skill** - it is not the job of the generic orchestrator, which produces a Markdown report.
|
|
42
|
+
|
|
43
|
+
## Phase 1 - Reconnaissance and project profile
|
|
44
|
+
|
|
45
|
+
Before launching the audits, **build a structured project profile** that will be passed to every subagent. This prevents each agent from rediscovering the context.
|
|
46
|
+
|
|
47
|
+
### Automatic detection
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
# Language and framework
|
|
51
|
+
ls package.json go.mod requirements.txt Cargo.toml pom.xml build.gradle 2>/dev/null
|
|
52
|
+
# CI/CD
|
|
53
|
+
ls .github/workflows/*.yml .gitlab-ci.yml Jenkinsfile bitbucket-pipelines.yml .circleci/config.yml 2>/dev/null
|
|
54
|
+
# Infrastructure
|
|
55
|
+
ls Dockerfile docker-compose*.yml terraform/ pulumi/ .cloud/ kubernetes/ helm/ 2>/dev/null
|
|
56
|
+
# Tests
|
|
57
|
+
ls vitest.config.* jest.config.* playwright.config.* cypress.config.* pytest.ini 2>/dev/null
|
|
58
|
+
# Quality
|
|
59
|
+
ls .eslintrc* eslint.config.* .prettierrc* biome.json .editorconfig tsconfig.json 2>/dev/null
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
### Profile to produce
|
|
63
|
+
|
|
64
|
+
Generate this text block and include it in EVERY subagent prompt:
|
|
65
|
+
|
|
66
|
+
```
|
|
67
|
+
## Project profile
|
|
68
|
+
- **Name**: [name]
|
|
69
|
+
- **Type**: [backend API | frontend SPA/PWA | fullstack | library/SDK | data pipeline | infrastructure]
|
|
70
|
+
- **Language(s)**: [TypeScript, Python, Go, etc.]
|
|
71
|
+
- **Framework**: [Nuxt, Next.js, NestJS, Django, Spring, etc.]
|
|
72
|
+
- **Architecture**: [monolith | monorepo | microservices | client-side only]
|
|
73
|
+
- **Backend**: [yes/no - if no, adapt the server-side questions]
|
|
74
|
+
- **Database**: [PostgreSQL, MongoDB, IndexedDB, none, etc.]
|
|
75
|
+
- **CI/CD**: [GitHub Actions | GitLab CI | Jenkins | none]
|
|
76
|
+
- **Deployment**: [Docker, K8s, serverless, static hosting, etc.]
|
|
77
|
+
- **Root path**: [/path/to/project]
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### Relevance matrix
|
|
81
|
+
|
|
82
|
+
Depending on the profile, mark the N/A domains:
|
|
83
|
+
|
|
84
|
+
| Project type | CI/CD | Testing | Security | Observability | Architecture | Quality | Compliance | Frontend Perf |
|
|
85
|
+
|---------------|-------|---------|----------|---------------|-------------|---------|------------|---------------|
|
|
86
|
+
| Backend API | yes | yes | yes | yes | yes | yes | yes | N/A |
|
|
87
|
+
| Frontend SPA/PWA | yes | yes | yes (adapted) | client-adapted | yes | yes | adapted | **yes** |
|
|
88
|
+
| Fullstack | yes | yes | yes | yes | yes | yes | yes | adapted |
|
|
89
|
+
| Library/SDK | yes | yes | adapted | N/A | adapted | yes | N/A | adapted |
|
|
90
|
+
| Data pipeline | yes | yes | yes | yes | adapted | yes | yes | N/A |
|
|
91
|
+
|
|
92
|
+
**"adapted"** = the domain applies but some questions become N/A. The subagent must read the "Variants" section of the matching skill.
|
|
93
|
+
|
|
94
|
+
## Phase 2 - Running the domain audits
|
|
95
|
+
|
|
96
|
+
Launch the **8 subagents in parallel** (Agent tool). Each subagent receives:
|
|
97
|
+
1. The **project profile** (block above)
|
|
98
|
+
2. The instruction to **read the domain's SKILL.md**
|
|
99
|
+
3. The expected **output format** (below)
|
|
100
|
+
|
|
101
|
+
### Prompt template for each subagent
|
|
102
|
+
|
|
103
|
+
```
|
|
104
|
+
You are auditing the [DOMAIN] domain on the following project.
|
|
105
|
+
|
|
106
|
+
[PROJECT PROFILE - paste the block]
|
|
107
|
+
|
|
108
|
+
## Instructions
|
|
109
|
+
1. Read the audit skill `audit-[domain]` (invoke it, or read its `SKILL.md`)
|
|
110
|
+
2. Follow the audit procedure described in the skill
|
|
111
|
+
3. For each question, assign a level (0-4) or N/A with:
|
|
112
|
+
- A factual justification (cite files, lines, commands)
|
|
113
|
+
- A confidence level (high/medium/low)
|
|
114
|
+
4. Run the verification commands listed in the skill whenever possible
|
|
115
|
+
5. Compute the domain score: average of the scored questions (exclude N/A)
|
|
116
|
+
|
|
117
|
+
## MANDATORY output format
|
|
118
|
+
|
|
119
|
+
### [Domain] - Score: X.X/4 (Y questions scored out of Z)
|
|
120
|
+
|
|
121
|
+
| Code | Question | Criticality | Level | Confidence | Justification |
|
|
122
|
+
|------|----------|-----------|--------|-----------|---------------|
|
|
123
|
+
| XX-01 | ... | must | X | high | ... |
|
|
124
|
+
| XX-02 | ... | should | N/A | - | ... |
|
|
125
|
+
|
|
126
|
+
### Strengths
|
|
127
|
+
- ...
|
|
128
|
+
|
|
129
|
+
### Weaknesses
|
|
130
|
+
- ...
|
|
131
|
+
|
|
132
|
+
### Recommendations
|
|
133
|
+
1. [MUST] ... *(XX-01: 0→2)*
|
|
134
|
+
2. [SHOULD] ... *(XX-03: 1→3)*
|
|
135
|
+
3. [COULD] ... *(XX-05: 2→3)*
|
|
136
|
+
|
|
137
|
+
### Non-auditable items
|
|
138
|
+
[Questions requiring an interview, field observation, or access to external systems]
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
## Phase 3 - Report consolidation
|
|
142
|
+
|
|
143
|
+
**Do not invoke audit-report.** Consolidate directly following the rules below.
|
|
144
|
+
|
|
145
|
+
### Score computation & classification
|
|
146
|
+
|
|
147
|
+
The domain weights, the domain/global score formula and the maturity classification are the
|
|
148
|
+
**single source of truth** owned by the `audit-report` skill (its `scoring-model.md`, under
|
|
149
|
+
that skill's `reference/` folder). Read that file and apply the exact weights and
|
|
150
|
+
thresholds - they are **not** duplicated here.
|
|
151
|
+
|
|
152
|
+
Reminders: domain score = mean of the scored questions (N/A excluded); global score =
|
|
153
|
+
weighted average of the audited domains (fully-N/A domains excluded); maturity levels run
|
|
154
|
+
from Initial to Optimized on the 0-4 CMMI-adapted grid.
|
|
155
|
+
|
|
156
|
+
### Recommendation prioritisation
|
|
157
|
+
|
|
158
|
+
Collect all recommendations from the 8 domains and sort them:
|
|
159
|
+
|
|
160
|
+
| Priority | Criterion |
|
|
161
|
+
|----------|-----------|
|
|
162
|
+
| **Immediate MUST** | Score 0 on a "must" criticality question |
|
|
163
|
+
| **MUST** | Score 0-1 on a "must" question, or active security risk |
|
|
164
|
+
| **SHOULD** | Score 0-1 on a "should" question, or score 1-2 on a "must" |
|
|
165
|
+
| **COULD** | Score 0-2 on a "could" question |
|
|
166
|
+
|
|
167
|
+
Group into:
|
|
168
|
+
1. **Quick wins** (effort < 1 day, immediate impact)
|
|
169
|
+
2. **Short term** (1-3 months)
|
|
170
|
+
3. **Medium term** (3-6 months)
|
|
171
|
+
4. **Vision** (6-12 months)
|
|
172
|
+
|
|
173
|
+
### Impact projection
|
|
174
|
+
|
|
175
|
+
For each action in the plan, indicate the expected progression: `(XX-01: current→projected)`.
|
|
176
|
+
Compute the projected score if quick wins + short term are implemented.
|
|
177
|
+
|
|
178
|
+
### Final report format
|
|
179
|
+
|
|
180
|
+
```markdown
|
|
181
|
+
# Software industrialisation audit report
|
|
182
|
+
|
|
183
|
+
**Project:** [name] | **Date:** [YYYY-MM-DD] | **Global score: X.X/4 - [Level]**
|
|
184
|
+
|
|
185
|
+
## Project profile
|
|
186
|
+
[Profile block]
|
|
187
|
+
|
|
188
|
+
## Executive summary
|
|
189
|
+
[3-5 sentences: overall state, major strengths, critical risks]
|
|
190
|
+
|
|
191
|
+
## Maturity radar
|
|
192
|
+
[Textual ASCII radar with the 8 domain scores (fully-N/A ones excluded)]
|
|
193
|
+
|
|
194
|
+
## Scores per domain
|
|
195
|
+
| Domain | Score | Questions | Level |
|
|
196
|
+
[table]
|
|
197
|
+
|
|
198
|
+
## Detail per domain
|
|
199
|
+
[For each domain: summary + question table + strengths/weaknesses]
|
|
200
|
+
|
|
201
|
+
## Action plan
|
|
202
|
+
### Quick wins
|
|
203
|
+
### Short term (1-3 months)
|
|
204
|
+
### Medium term (3-6 months)
|
|
205
|
+
### Vision (6-12 months)
|
|
206
|
+
|
|
207
|
+
## Impact projection
|
|
208
|
+
[Current vs projected table per domain]
|
|
209
|
+
|
|
210
|
+
## Non-auditable items
|
|
211
|
+
[Questions requiring an interview, an observation, or external access]
|
|
212
|
+
|
|
213
|
+
## Appendices
|
|
214
|
+
### Validation methods
|
|
215
|
+
### Reference model
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
Save to `docs/audit/YYYY-MM-DD-audit-industrialisation.md`.
|
|
219
|
+
|
|
220
|
+
## Phase 4 - Evidence & traceability
|
|
221
|
+
|
|
222
|
+
**The auditor does not give an opinion - it proves.** Every assigned level rests on factual evidence, captured during Phase 2 and cited in the report:
|
|
223
|
+
|
|
224
|
+
- **`code_analysis`**: excerpt of the relevant source file
|
|
225
|
+
- **`technical_test`**: output of an executed command (grep, `npm audit`, `ls`…)
|
|
226
|
+
- **`document`**: excerpt of a reference document (README, ADR…)
|
|
227
|
+
|
|
228
|
+
Rules:
|
|
229
|
+
- Cite the evidence next to the justification (`questionId` + short description).
|
|
230
|
+
- Capture command outputs rather than summarising them from memory.
|
|
231
|
+
- No evidence possible → question `N/A` / non-auditable with its reason, never scored blind.
|
|
232
|
+
- Keep, per question: level (0-4 or N/A), criticality (must/should/could), confidence (high/medium/low), validation method. These fields feed the report's "Detail per domain" table and the "Validation methods" appendix.
|
|
233
|
+
|
|
234
|
+
## Adaptation
|
|
235
|
+
|
|
236
|
+
If `$ARGUMENTS` contains domain names (`ci`, `testing`, `security`, `observability`, `architecture`, `quality`, `compliance`, `performance-frontend`), only launch the requested domains. The report adapts to the scope.
|
|
237
|
+
|
|
238
|
+
## Exit condition
|
|
239
|
+
|
|
240
|
+
- [ ] Project profile built and passed to each subagent (Phase 1)
|
|
241
|
+
- [ ] Relevance matrix applied - N/A domains marked, not audited blind
|
|
242
|
+
- [ ] 8 domain audits run (or requested subset), each question scored 0-4 or N/A **with factual evidence**
|
|
243
|
+
- [ ] Weighted global score computed (N/A excluded), maturity classification assigned
|
|
244
|
+
- [ ] Prioritised action plan (MUST/SHOULD/COULD) + impact projection
|
|
245
|
+
- [ ] Markdown report saved to `docs/audit/`; evidence cited per question
|
|
246
|
+
|
|
247
|
+
## Changelog
|
|
248
|
+
|
|
249
|
+
- 1.0.1 (2026-07-20) - audit-report invocation wording fixed (never invoked, template reused); performance-frontend added to $ARGUMENTS domains
|
|
250
|
+
|
|
251
|
+
- 1.0.0 (2026-06-19) - initial versioned release + state-of-the-art enrichment (routing, context, protocol, traps, exit condition)
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# `audit-observability`
|
|
2
|
+
|
|
3
|
+
> Score a project's observability and alerting maturity - logs, metrics, traces, dashboards, alerts - on a 0-4 scale across 15 questions, with evidence gathered from the code itself.
|
|
4
|
+
|
|
5
|
+
| | |
|
|
6
|
+
|---|---|
|
|
7
|
+
| **Type** | Skill (on-demand audit) |
|
|
8
|
+
| **Category** | `audit` |
|
|
9
|
+
| **Artifact** | [`SKILL.md`](SKILL.md) |
|
|
10
|
+
| **Tooling** | Grep/find verification commands; progressive-disclosure grids in [`reference/`](reference/) |
|
|
11
|
+
|
|
12
|
+
## What it is
|
|
13
|
+
|
|
14
|
+
`audit-observability` is a structured maturity assessment for one domain: **can you see what your system is doing, and are you told when it breaks?** It walks 15 questions split into two areas - Observability (OB, 9 questions: logging, baggage, metrics, traces, dashboards, latency, pools, transient errors) and Alerting (AL, 6 questions: catalog, notifications, correlation, runbooks, thresholds, post-mortems) - and assigns each a level from 0 (absent) to 4 (state of the art), backed by concrete evidence from the repository.
|
|
15
|
+
|
|
16
|
+
The heavy detail lives in four reference grids, loaded on demand:
|
|
17
|
+
|
|
18
|
+
- [`reference/logging.md`](reference/logging.md) - ob-01 to ob-03
|
|
19
|
+
- [`reference/metriques.md`](reference/metriques.md) - ob-04, ob-07, ob-08, ob-09
|
|
20
|
+
- [`reference/traces-dashboards.md`](reference/traces-dashboards.md) - ob-05, ob-06
|
|
21
|
+
- [`reference/alerting.md`](reference/alerting.md) - al-01 to al-06
|
|
22
|
+
|
|
23
|
+
## Why it exists
|
|
24
|
+
|
|
25
|
+
Observability gaps are invisible until an incident, and by then it is too late to add instrumentation. A maturity audit surfaces the gaps in advance, prioritized: which MUST-level controls are missing, which SHOULD-level practices would cut mean-time-to-detect, and which are simply not applicable to this project type. Turning a vague "we should have better monitoring" into a scored, evidence-backed grid makes the work plannable.
|
|
26
|
+
|
|
27
|
+
It is deliberately **single-domain**. For a whole-project maturity review across all engineering domains, `audit-industrialisation` orchestrates this skill together with its siblings and consolidates the scores - invoke that instead of running each audit by hand.
|
|
28
|
+
|
|
29
|
+
## When it triggers
|
|
30
|
+
|
|
31
|
+
Invoke it when the user asks for a targeted observability or alerting review:
|
|
32
|
+
|
|
33
|
+
- "audit observability" / "observability audit"
|
|
34
|
+
- "audit logs" / "audit monitoring" / "audit alerting"
|
|
35
|
+
|
|
36
|
+
Do **not** invoke it for a full multi-domain audit - route to `audit-industrialisation`, which calls this skill as one of its components.
|
|
37
|
+
|
|
38
|
+
## How it works
|
|
39
|
+
|
|
40
|
+
The skill detects the project type (to pick the grid variant and N/A set) and the observability stack, then works each of the 15 questions from its reference grid - running the verification commands, reading the evidence, and assigning a level 0-4 with a confidence and justification - before averaging the scored questions and emitting the report.
|
|
41
|
+
|
|
42
|
+
Full step-by-step protocol → [`SKILL.md`](SKILL.md).
|
|
43
|
+
|
|
44
|
+
### Scoring at a glance
|
|
45
|
+
|
|
46
|
+
Each question carries a criticality (MUST / SHOULD / COULD) that drives recommendation priority, and a level table where roughly: 0 = absent, 1 = ad-hoc, 2 = basic/documented, 3 = systematic with tooling, 4 = state of the art (ML, adaptive, self-service). The overall score is a plain average of scored questions - N/A never drags it down.
|
|
47
|
+
|
|
48
|
+
## Worked example
|
|
49
|
+
|
|
50
|
+
> A backend API is audited. Logs are JSON with a timestamp and level but carry no traceId; there is a Prometheus `/metrics` endpoint with RED metrics on the main routes; there is no runbook and no post-mortem template.
|
|
51
|
+
|
|
52
|
+
- **ob-01 (structured logs, MUST)** → level 2. JSON with basic fields but no traceId/spanId for cross-service correlation. Confidence: high (grep found the logger config and no `traceparent`).
|
|
53
|
+
- **ob-04 (RED/USE, SHOULD)** → level 2. RED on main endpoints, no USE on resources. Confidence: high.
|
|
54
|
+
- **al-04 (runbook, SHOULD)** → level 0. `find` returned no runbook/playbook file. Confidence: high.
|
|
55
|
+
- **al-06 (post-mortem, SHOULD)** → level 0. No template found.
|
|
56
|
+
|
|
57
|
+
The report then recommends, in priority order: `[MUST]` add traceId propagation to reach ob-01 level 3, `[SHOULD]` introduce a runbook per critical alert, `[SHOULD]` adopt a blameless post-mortem template.
|
|
58
|
+
|
|
59
|
+
## Related artifacts
|
|
60
|
+
|
|
61
|
+
- [`audit-industrialisation`](../audit-industrialisation/) - orchestrates all domain audits (including this one) into a consolidated report; use it for a full project audit.
|
|
62
|
+
- [`audit-report`](../audit-report/) - template and scoring rules for the consolidated report.
|
|
63
|
+
- [`audit-quality`](../audit-quality/) - QA/DevOps audit; overlaps on error handling and observability from the quality angle.
|
|
64
|
+
- [`audit-architecture`](../audit-architecture/) - resilience, SLA/RPO/RTO and incident management, which pair naturally with alerting maturity.
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: audit-observability
|
|
3
|
+
description: Observability & Alerting audit. Use when auditing logging, metrics, tracing, dashboards, or alerting. Triggers on "audit observabilité", "audit observability", "observability audit", "audit logs", "audit monitoring", "audit alertes", "audit alerting".
|
|
4
|
+
version: 1.0.0
|
|
5
|
+
metadata:
|
|
6
|
+
last-reviewed: 2026-06-19
|
|
7
|
+
category: audit
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## When ME and not audit-industrialisation
|
|
11
|
+
|
|
12
|
+
- **ME** when: audit targeted at this domain only
|
|
13
|
+
- **audit-industrialisation** instead if: global multi-domain audit - it orchestrates every audit (including ME) and produces a consolidated report
|
|
14
|
+
|
|
15
|
+
> For a full project audit, use `audit-industrialisation` rather than invoking each skill separately.
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
# Observability & Alerting Audit
|
|
20
|
+
|
|
21
|
+
## Overview
|
|
22
|
+
|
|
23
|
+
Audit a project's observability maturity: structured logs, logging guidelines, contextual baggage, metrics, dashboards, distributed traces, latency tracking, pool saturation, transient-error detection, alerting, notifications, correlation, runbooks, threshold tuning and post-mortems. Covers the OB (Observability, 9 questions) and AL (Alerting, 6 questions) sections, i.e. **15 questions** across 4 thematic grids. Produces a maturity score (0-4) per question.
|
|
24
|
+
|
|
25
|
+
## Score calculation
|
|
26
|
+
|
|
27
|
+
- Each scored question receives a level from 0 to 4.
|
|
28
|
+
- Questions marked **N/A** are excluded from the calculation.
|
|
29
|
+
- **Overall score** = sum of levels / number of scored questions (excluding N/A).
|
|
30
|
+
- Formula: `score = Σ(levels) / count(scored questions)`
|
|
31
|
+
- Example: 10 scored questions out of 15, sum = 25 → score = 25/10 = 2.5/4
|
|
32
|
+
|
|
33
|
+
## Audit protocol
|
|
34
|
+
|
|
35
|
+
1. **Detect the project type**: backend, frontend SPA/PWA, fullstack, library. Use the client-side variant if applicable.
|
|
36
|
+
2. **Detect the observability stack**: look for OpenTelemetry, Prometheus, Grafana, ELK, Datadog, Sentry configs
|
|
37
|
+
3. **Analyze the logging configuration**: framework, format, levels, correlation, guidelines
|
|
38
|
+
4. **Check contextual baggage**: context propagation, automatic enrichment
|
|
39
|
+
5. **Check instrumentation**: RED/USE metrics exposed, tracing configured
|
|
40
|
+
6. **Look for dashboards**: Grafana configs, links in the docs, hierarchical structure
|
|
41
|
+
7. **Analyze latency tracking**: baseline, drift detection, predictive alerts
|
|
42
|
+
8. **Check pools**: configurations, metrics, saturation alerts
|
|
43
|
+
9. **Examine transient-error detection**: classification, baseline, anomalies
|
|
44
|
+
10. **Examine the alert catalog**: classification, owner, runbook, severity
|
|
45
|
+
11. **Check notifications**: channels, routing, escalation, failover
|
|
46
|
+
12. **Analyze correlation/deduplication**: grouping, inhibition, noise ratio
|
|
47
|
+
13. **Check runbooks**: content, links, diagnostic scripts
|
|
48
|
+
14. **Analyze threshold tuning**: regular review, metrics, adaptive thresholds
|
|
49
|
+
15. **Examine post-mortems**: template, culture, action tracking, recurrence
|
|
50
|
+
16. **Check dependencies**: observability packages installed
|
|
51
|
+
17. **Run the verification commands** for each question (see per-section grids)
|
|
52
|
+
18. **Assign a level** per question with justification and confidence level
|
|
53
|
+
19. **Mark N/A** the questions not applicable to the project type
|
|
54
|
+
20. **Produce the report**
|
|
55
|
+
|
|
56
|
+
## Per-section grids
|
|
57
|
+
|
|
58
|
+
Each grid contains the statement, criticality, the elements to analyze/check, the bash commands and the level tables 0-4 (with a client-side variant where relevant).
|
|
59
|
+
|
|
60
|
+
- [reference/logging.md](reference/logging.md) - Logs, guidelines, contextual baggage · **3 questions** (ob-01 to ob-03)
|
|
61
|
+
- [reference/metriques.md](reference/metriques.md) - RED/USE, latency, pool saturation, transient errors · **4 questions** (ob-04, ob-07, ob-08, ob-09)
|
|
62
|
+
- [reference/traces-dashboards.md](reference/traces-dashboards.md) - Real-time dashboards, distributed traces · **2 questions** (ob-05, ob-06)
|
|
63
|
+
- [reference/alerting.md](reference/alerting.md) - Catalog, notifications, correlation, runbooks, thresholds, post-mortems · **6 questions** (al-01 to al-06)
|
|
64
|
+
|
|
65
|
+
### Variants by project type
|
|
66
|
+
|
|
67
|
+
| Project type | Variant to use | Often-N/A questions |
|
|
68
|
+
|----------------|--------------------|-----------------------|
|
|
69
|
+
| **Backend / API** | Standard grid (server) | - |
|
|
70
|
+
| **Frontend SPA/PWA** | Client-side grid for all questions | ob-03, ob-08 may stay N/A |
|
|
71
|
+
| **Fullstack (monorepo)** | Standard for the backend, client-side for the frontend. Two scores possible. | - |
|
|
72
|
+
| **Library / SDK** | ob-01 (logs) and ob-02 (guidelines) relevant, the rest often N/A | ob-04 to ob-09, al-01 to al-06 often N/A |
|
|
73
|
+
| **CLI / Script** | Simplified standard | ob-03, ob-05, ob-06, ob-08, al-02 to al-06 often N/A |
|
|
74
|
+
|
|
75
|
+
### Output format
|
|
76
|
+
|
|
77
|
+
```markdown
|
|
78
|
+
## Observability & Alerting - Overall score: X.X/4 (Y questions scored out of 15)
|
|
79
|
+
|
|
80
|
+
### Summary
|
|
81
|
+
[2-3 sentences summarizing observability maturity]
|
|
82
|
+
[Detected project type: backend / frontend SPA/PWA / fullstack / library]
|
|
83
|
+
|
|
84
|
+
### Detail per question
|
|
85
|
+
|
|
86
|
+
| Code | Question | Criticality | Level | Confidence | Justification |
|
|
87
|
+
|------|----------|-----------|--------|-----------|---------------|
|
|
88
|
+
| ob-01 | Correlated structured logs | MUST | X | high/medium/low | ... |
|
|
89
|
+
| ob-02 | Logging guideline and message structure | SHOULD | X | high/medium/low | ... |
|
|
90
|
+
| ob-03 | Analysis and implementation of useful baggage | SHOULD | X | high/medium/low | ... |
|
|
91
|
+
| ob-04 | RED/USE metrics | SHOULD | X | high/medium/low | ... |
|
|
92
|
+
| ob-05 | Real-time dashboards | SHOULD | X | high/medium/low | ... |
|
|
93
|
+
| ob-06 | Distributed traces (OTel, Elastic APM) | COULD | X or N/A | high/medium/low | ... |
|
|
94
|
+
| ob-07 | Latency drift tracking | SHOULD | X | high/medium/low | ... |
|
|
95
|
+
| ob-08 | Pool saturation (thread, DB, API) | COULD | X or N/A | high/medium/low | ... |
|
|
96
|
+
| ob-09 | Abnormal transient-error detection | SHOULD | X or N/A | high/medium/low | ... |
|
|
97
|
+
| al-01 | Alert catalog (critical, warning, info) | MUST | X | high/medium/low | ... |
|
|
98
|
+
| al-02 | Multi-channel notifications | SHOULD | X | high/medium/low | ... |
|
|
99
|
+
| al-03 | Correlation/deduplication (alert fatigue) | SHOULD | X | high/medium/low | ... |
|
|
100
|
+
| al-04 | Investigation runbook | SHOULD | X | high/medium/low | ... |
|
|
101
|
+
| al-05 | Alert threshold tuning | SHOULD | X | high/medium/low | ... |
|
|
102
|
+
| al-06 | Post-mortem template + recurring incidents | SHOULD | X or N/A | high/medium/low | ... |
|
|
103
|
+
|
|
104
|
+
### Strengths
|
|
105
|
+
- ...
|
|
106
|
+
|
|
107
|
+
### Weaknesses
|
|
108
|
+
- ...
|
|
109
|
+
|
|
110
|
+
### Recommendations
|
|
111
|
+
1. [MUST] ...
|
|
112
|
+
2. [SHOULD] ...
|
|
113
|
+
3. [COULD] ...
|
|
114
|
+
|
|
115
|
+
### Non-auditable items
|
|
116
|
+
- [List of items impossible to verify through static code analysis]
|
|
117
|
+
- Examples: SaaS-side Sentry configuration, alerting rules in PagerDuty, hosted Grafana dashboards, SLOs defined outside the repo, post-mortem process, notification tests
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
## Exit condition
|
|
121
|
+
|
|
122
|
+
The audit is complete when:
|
|
123
|
+
|
|
124
|
+
- [ ] Project type detected (backend / frontend / fullstack / library / CLI), variant chosen
|
|
125
|
+
- [ ] All 15 questions have received a level 0-4 or been marked N/A with justification
|
|
126
|
+
- [ ] Verification commands run for each scored question
|
|
127
|
+
- [ ] Confidence level (high/medium/low) assigned per question
|
|
128
|
+
- [ ] Overall score computed excluding N/A
|
|
129
|
+
- [ ] Strengths, weaknesses and prioritized recommendations (MUST/SHOULD/COULD) filled in
|
|
130
|
+
- [ ] Non-auditable items listed explicitly
|
|
131
|
+
|
|
132
|
+
## Changelog
|
|
133
|
+
|
|
134
|
+
- 1.0.0 (2026-06-19) - initial versioned release + state-of-the-art enrichment (routing, context, protocol, traps, exit condition); audit grids in reference/ (progressive disclosure)
|
|
@@ -0,0 +1,214 @@
|
|
|
1
|
+
# Alerting section - Catalog, notifications, correlation, runbooks, thresholds, post-mortems
|
|
2
|
+
|
|
3
|
+
6 questions. Alert catalog, multi-channel notifications, correlation/deduplication, runbooks, threshold tuning, post-mortems.
|
|
4
|
+
|
|
5
|
+
## Table of contents
|
|
6
|
+
|
|
7
|
+
- [al-01 - Alert catalog (critical, warning, info)](#al-01--alert-catalog-critical-warning-info--criticality-must)
|
|
8
|
+
- [al-02 - Multi-channel notifications](#al-02--multi-channel-notifications--criticality-should)
|
|
9
|
+
- [al-03 - Correlation/deduplication (alert fatigue)](#al-03--correlationdeduplication-alert-fatigue--criticality-should)
|
|
10
|
+
- [al-04 - Investigation runbook](#al-04--investigation-runbook--criticality-should)
|
|
11
|
+
- [al-05 - Alert threshold tuning](#al-05--alert-threshold-tuning--criticality-should)
|
|
12
|
+
- [al-06 - Post-mortem template + recurring incidents](#al-06--post-mortem-template--recurring-incidents--criticality-should)
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
### al-01 - Alert catalog (critical, warning, info) · Criticality: **MUST**
|
|
17
|
+
|
|
18
|
+
**Analyze:** Alert configs (Alertmanager, PagerDuty, OpsGenie, Grafana alerts), alert rules, severity classification, alerting-as-code
|
|
19
|
+
|
|
20
|
+
**Verification commands:**
|
|
21
|
+
```bash
|
|
22
|
+
# Check for the presence of alerting tools
|
|
23
|
+
grep -ri "sentry\|bugsnag\|logrocket\|pagerduty\|opsgenie\|alertmanager\|grafana.*alert" package.json composer.json docker-compose*.yml 2>/dev/null
|
|
24
|
+
# Look for alert configuration files
|
|
25
|
+
find . -name "*alert*" -o -name "*rules*" -o -name "*.rules.yml" 2>/dev/null | head -10
|
|
26
|
+
# Look for alert classification
|
|
27
|
+
grep -ri "critical\|warning\|info\|severity\|priority\|P1\|P2\|P3\|P4" --include="*.yaml" --include="*.yml" --include="*.rules" 2>/dev/null | head -10
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
**Check:**
|
|
31
|
+
- Structured catalog with classification: Critical (immediate action), Warning (action within 1h), Info (to consult)
|
|
32
|
+
- Owner defined per alert (person or team responsible)
|
|
33
|
+
- Trigger conditions documented (PromQL expression, threshold, frequency)
|
|
34
|
+
- Alerting-as-code (versioned configs, reviewed in code review)
|
|
35
|
+
- Quarterly review of the catalog (pruning obsolete alerts)
|
|
36
|
+
- Runbook linked to each alert
|
|
37
|
+
|
|
38
|
+
**Levels:**
|
|
39
|
+
| Level | Criteria |
|
|
40
|
+
|--------|----------|
|
|
41
|
+
| 0 | No catalog. Alerts created on the fly without consistency. Duplicates and gaps. |
|
|
42
|
+
| 1 | Informal list of alerts. No severity classification. Confusion about who responds. |
|
|
43
|
+
| 2 | Basic catalog with 3 levels (crit/warn/info). Partial documentation of conditions. |
|
|
44
|
+
| 3 | Complete catalog with owner, runbook, response SLA per alert. Quarterly review. |
|
|
45
|
+
| 4 | Living catalog with effectiveness metrics. Auto-documentation from config. Quality scoring. |
|
|
46
|
+
|
|
47
|
+
**Client-side variant (SPA/PWA):**
|
|
48
|
+
| Level | Criteria |
|
|
49
|
+
|--------|----------|
|
|
50
|
+
| 0 | No alerting. Errors discovered by users. |
|
|
51
|
+
| 1 | Email on crash (via Sentry or equivalent, no fine configuration). |
|
|
52
|
+
| 2 | Sentry alerting configured (thresholds, Slack/Teams notifications). |
|
|
53
|
+
| 3 | PagerDuty/Slack on critical errors. Alerts per feature/page. Severity defined. |
|
|
54
|
+
| 4 | SLO-based client alerting. Burn rate on error rates. Alert fatigue under control. |
|
|
55
|
+
|
|
56
|
+
---
|
|
57
|
+
|
|
58
|
+
### al-02 - Multi-channel notifications · Criticality: **SHOULD**
|
|
59
|
+
|
|
60
|
+
**Analyze:** Configured notification channels, routing by severity, failover, automatic escalation
|
|
61
|
+
|
|
62
|
+
**Verification commands:**
|
|
63
|
+
```bash
|
|
64
|
+
# Check the configured notification channels
|
|
65
|
+
grep -ri "slack\|email\|sms\|pagerduty\|opsgenie\|teams\|webhook\|notification.*channel" --include="*.yaml" --include="*.yml" --include="*.env*" 2>/dev/null | head -10
|
|
66
|
+
# Look for routing/escalation configuration
|
|
67
|
+
grep -ri "escalat\|routing\|receiver\|notify\|on_call\|schedule" --include="*.yaml" --include="*.yml" 2>/dev/null | head -10
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
**Check:**
|
|
71
|
+
- Redundant channels (Slack + email + SMS + call for critical ones)
|
|
72
|
+
- Routing by severity: Critical → call + SMS, Warning → Slack + email, Info → Slack only
|
|
73
|
+
- Automatic failover if no acknowledgment (PagerDuty/OpsGenie escalation)
|
|
74
|
+
- Schedule handling (business hours vs on-call)
|
|
75
|
+
- Regular delivery tests on all channels
|
|
76
|
+
- Heartbeat checks on notification channels
|
|
77
|
+
|
|
78
|
+
**Levels:**
|
|
79
|
+
| Level | Criteria |
|
|
80
|
+
|--------|----------|
|
|
81
|
+
| 0 | Single channel (email or Slack). If unavailable, alerts are lost. |
|
|
82
|
+
| 1 | 2 channels but no automatic fallback. Manual configuration. |
|
|
83
|
+
| 2 | Multi-channel (Slack, email, SMS) with routing by severity. No fallback test. |
|
|
84
|
+
| 3 | Automatic failover between channels. Regular delivery tests. Acknowledgment required for critical ones. |
|
|
85
|
+
| 4 | Smart orchestration (PagerDuty/OpsGenie). Automatic escalation. Response-time metrics. |
|
|
86
|
+
|
|
87
|
+
---
|
|
88
|
+
|
|
89
|
+
### al-03 - Correlation/deduplication (alert fatigue) · Criticality: **SHOULD**
|
|
90
|
+
|
|
91
|
+
**Analyze:** Alert deduplication, grouping, cross-service correlation, noise suppression, noise-ratio metrics
|
|
92
|
+
|
|
93
|
+
**Verification commands:**
|
|
94
|
+
```bash
|
|
95
|
+
# Look for Alertmanager grouping/inhibition configuration
|
|
96
|
+
grep -ri "group_by\|group_wait\|group_interval\|inhibit_rules\|deduplic\|fingerprint" --include="*.yaml" --include="*.yml" 2>/dev/null | head -10
|
|
97
|
+
# Look for references to alert fatigue or correlation
|
|
98
|
+
grep -ri "alert.*fatigue\|noise\|correlat\|dedup\|suppress\|inhibit" --include="*.yaml" --include="*.yml" --include="*.md" 2>/dev/null | head -5
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
**Check:**
|
|
102
|
+
- Deduplication by fingerprint (identical alerts merged)
|
|
103
|
+
- Alert grouping (per service, tenant, time window)
|
|
104
|
+
- Cross-service correlation (root-cause identification)
|
|
105
|
+
- Inhibition rules (if node down → suppress pod alerts)
|
|
106
|
+
- Noise-ratio metrics (< 20% non-actionable alerts)
|
|
107
|
+
- Alert volume per day per person (< 50-100)
|
|
108
|
+
|
|
109
|
+
**Levels:**
|
|
110
|
+
| Level | Criteria |
|
|
111
|
+
|--------|----------|
|
|
112
|
+
| 0 | No deduplication. Alert storm possible. Chronic alert fatigue. |
|
|
113
|
+
| 1 | Basic throttling (max N alerts/hour). No smart correlation. |
|
|
114
|
+
| 2 | Deduplication by fingerprint. Grouping of similar alerts. |
|
|
115
|
+
| 3 | Cross-service correlation. Root-cause identification. Suppression of symptomatic alerts. |
|
|
116
|
+
| 4 | ML for correlated anomaly detection. Auto-tuning of thresholds. Noise reduction >80%. |
|
|
117
|
+
|
|
118
|
+
---
|
|
119
|
+
|
|
120
|
+
### al-04 - Investigation runbook · Criticality: **SHOULD**
|
|
121
|
+
|
|
122
|
+
**Analyze:** Documented runbooks, direct link from alerts, automated diagnostic scripts, post-incident feedback loop
|
|
123
|
+
|
|
124
|
+
**Verification commands:**
|
|
125
|
+
```bash
|
|
126
|
+
# Look for runbooks in the repo
|
|
127
|
+
find . -name "*runbook*" -o -name "*playbook*" -o -name "*incident*" -o -name "*troubleshoot*" 2>/dev/null | head -10
|
|
128
|
+
# Look for runbook links in alert configs
|
|
129
|
+
grep -ri "runbook_url\|runbook\|playbook\|documentation_url" --include="*.yaml" --include="*.yml" --include="*.rules" 2>/dev/null | head -10
|
|
130
|
+
# Look for diagnostic scripts
|
|
131
|
+
find . -name "*diagnos*" -o -name "*debug*" -o -name "*healthcheck*" | grep -i script 2>/dev/null | head -5
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
**Check:**
|
|
135
|
+
- Runbooks documented for critical alerts (decision tree)
|
|
136
|
+
- Standard content: context, prerequisites, diagnostic steps, exact commands, decision criteria, remediation, escalation
|
|
137
|
+
- Direct link from the alert (runbook_url annotation)
|
|
138
|
+
- Automated diagnostic scripts (collection of contextual info)
|
|
139
|
+
- Feedback loop: post-incident update
|
|
140
|
+
- Resolution metrics (runbook coverage rate, accuracy score)
|
|
141
|
+
|
|
142
|
+
**Levels:**
|
|
143
|
+
| Level | Criteria |
|
|
144
|
+
|--------|----------|
|
|
145
|
+
| 0 | No runbook. Improvised investigation on each alert. |
|
|
146
|
+
| 1 | A few informal notes. Knowledge in the experts' heads. |
|
|
147
|
+
| 2 | Runbooks documented for critical alerts. Variable format. Not always up to date. |
|
|
148
|
+
| 3 | Structured runbook per alert with steps, commands, escalation. Direct link from the alert. |
|
|
149
|
+
| 4 | Interactive runbooks with partial auto-remediation. Resolution metrics. Feedback loop. |
|
|
150
|
+
|
|
151
|
+
---
|
|
152
|
+
|
|
153
|
+
### al-05 - Alert threshold tuning · Criticality: **SHOULD**
|
|
154
|
+
|
|
155
|
+
**Analyze:** Threshold review process, false-positive metrics, adaptive thresholds, SLO-based alerting
|
|
156
|
+
|
|
157
|
+
**Verification commands:**
|
|
158
|
+
```bash
|
|
159
|
+
# Look for thresholds in alert configs
|
|
160
|
+
grep -ri "threshold\|seuil\|baseline\|for:\|expr:" --include="*.yaml" --include="*.yml" --include="*.rules" 2>/dev/null | head -10
|
|
161
|
+
# Look for references to SLOs
|
|
162
|
+
grep -ri "slo\|error.*budget\|burn.*rate\|service.*level" --include="*.yaml" --include="*.yml" --include="*.md" --include="*.ts" 2>/dev/null | head -10
|
|
163
|
+
# Look for adaptive thresholds
|
|
164
|
+
grep -ri "anomaly\|predict_linear\|forecast\|dynamic.*threshold\|adaptive" --include="*.yaml" --include="*.yml" --include="*.rules" 2>/dev/null | head -5
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
**Check:**
|
|
168
|
+
- Regular threshold review (quarterly minimum)
|
|
169
|
+
- Objective metrics: false positive rate, alert-to-incident ratio, MTTR per alert type
|
|
170
|
+
- Adaptive vs static thresholds (dynamic baseline)
|
|
171
|
+
- SLO-based approach (alerts on burn rate rather than raw thresholds)
|
|
172
|
+
- A/B testing of new alert configurations
|
|
173
|
+
- Documentation of thresholds and their justification
|
|
174
|
+
|
|
175
|
+
**Levels:**
|
|
176
|
+
| Level | Criteria |
|
|
177
|
+
|--------|----------|
|
|
178
|
+
| 0 | Default thresholds never adjusted. Many false positives or gaps. |
|
|
179
|
+
| 1 | Reactive adjustment after major incidents. No process. |
|
|
180
|
+
| 2 | Annual threshold review. Based on team feedback. No objective data. |
|
|
181
|
+
| 3 | Quarterly review with metrics (false positive rate, MTTR). Thresholds documented. |
|
|
182
|
+
| 4 | Adaptive thresholds based on dynamic baseline. A/B testing of configurations. Continuous optimization. |
|
|
183
|
+
|
|
184
|
+
---
|
|
185
|
+
|
|
186
|
+
### al-06 - Post-mortem template + recurring incidents · Criticality: **SHOULD**
|
|
187
|
+
|
|
188
|
+
**Analyze:** Post-mortem template, blameless culture, corrective-action tracking, recurrence metrics
|
|
189
|
+
|
|
190
|
+
**Verification commands:**
|
|
191
|
+
```bash
|
|
192
|
+
# Look for post-mortem templates or reports
|
|
193
|
+
find . -name "*postmortem*" -o -name "*post-mortem*" -o -name "*post_mortem*" -o -name "*incident-report*" -o -name "*retrospective*" 2>/dev/null | head -10
|
|
194
|
+
# Look for references to the post-mortem process in the docs
|
|
195
|
+
grep -ri "post.mortem\|blameless\|root.cause\|incident.*review\|lessons.*learned" --include="*.md" --include="*.yaml" --include="*.yml" 2>/dev/null | head -10
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
**Check:**
|
|
199
|
+
- Standardized template: summary, timeline, impact, root cause, contributing factors, corrective actions, lessons
|
|
200
|
+
- Blameless culture (focus on the system, not people)
|
|
201
|
+
- Systematic post-mortem for P1/P2 incidents
|
|
202
|
+
- Review board (collective validation of the analyses)
|
|
203
|
+
- Actions integrated into the backlog with owner and deadline
|
|
204
|
+
- Recurrence metrics (similar incidents after post-mortem)
|
|
205
|
+
- Cross-team sharing of learnings
|
|
206
|
+
|
|
207
|
+
**Levels:**
|
|
208
|
+
| Level | Criteria |
|
|
209
|
+
|--------|----------|
|
|
210
|
+
| 0 | No post-mortem. The same incidents keep recurring. |
|
|
211
|
+
| 1 | Occasional post-mortem for major incidents. Free format. Actions rarely followed up. |
|
|
212
|
+
| 2 | Standardized post-mortem template. Done for P1/P2 incidents. Actions tracked. |
|
|
213
|
+
| 3 | Systematic blameless post-mortem. Review board. Actions integrated into the backlog with priority. |
|
|
214
|
+
| 4 | Strong post-mortem culture. Cross-team sharing. Recurrence metrics. Proven improvement. |
|