fdeops 3.11.1 → 3.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +1 -1
- package/README.md +189 -129
- package/adapters/AGENTS.md +4 -1
- package/adapters/GEMINI.md +4 -1
- package/adapters/LOCAL-LLM.md +1 -1
- package/adapters/README.md +2 -2
- package/adapters/copilot-instructions.md +4 -1
- package/adapters/cursor.fde.mdc +4 -2
- package/bin/check.js +74 -25
- package/bin/fde.js +92 -43
- package/bin/install.js +1 -1
- package/bin/lib/trust.js +3 -2
- package/mcp/fdeops-ingest/package.json +1 -1
- package/package.json +5 -2
- package/plugin.json +3 -3
- package/skills/fde/SKILL.md +80 -231
- package/skills/fde/references/pattern-extract.md +9 -9
- package/skills/fde/references/status.md +2 -0
- package/skills/fde/references/build.md +0 -154
- package/skills/fde/references/debug.md +0 -55
- package/skills/fde/references/observability.md +0 -103
- package/skills/fde/references/qa-live.md +0 -113
- package/skills/fde/references/security-audit.md +0 -105
- package/skills/fde/references/test-on-legacy.md +0 -108
|
@@ -1,154 +0,0 @@
|
|
|
1
|
-
# build - safe implementation on someone else's codebase
|
|
2
|
-
|
|
3
|
-
**Enter when:** an agreed slice is ready to implement - feature, fix, integration, legacy change. Includes the review gate before merge.
|
|
4
|
-
|
|
5
|
-
**Read first:** `context.md`, `terrain.md`, `decisions.md`. Load `trust-profile.md` when touching regulated areas. No map or no plan → route to discover/plan first; say it plainly: "We're not ready to touch code until we know what's connected to this module."
|
|
6
|
-
|
|
7
|
-
## Validation gate (confirm understanding, clarify where it elevates)
|
|
8
|
-
|
|
9
|
-
Before starting, state what you're working with in 2-4 lines - a brief playback that invites correction, not a question:
|
|
10
|
-
|
|
11
|
-
> "Building: [task name]. Blast radius: [files/systems]. Terrain is [X days] fresh. [Any risk or concern worth naming, or 'clear to proceed']."
|
|
12
|
-
|
|
13
|
-
Then check - probe ONLY if it prevents a mistake:
|
|
14
|
-
|
|
15
|
-
1. **Acceptance criteria are falsifiable.** If the criteria are vague ("should work well", "handle errors gracefully") → rephrase them specifically: "I'm reading 'works well' as: responds in <500ms, retries 3x, alerts on failure. That right?"
|
|
16
|
-
2. **Terrain is current.** If terrain.md is significantly older than the plan → one line: "Terrain is from Day 3, plan from Day 8 - assuming nothing shifted in between."
|
|
17
|
-
3. **No open CRITICAL risk on the module.** If found → name it: "There's an open risk on this module (rate limiting). Building around it unless you say otherwise."
|
|
18
|
-
|
|
19
|
-
Don't interrogate. State your read, let the FDE correct if needed, then move.
|
|
20
|
-
|
|
21
|
-
## Spec generation (AI generates, human approves)
|
|
22
|
-
|
|
23
|
-
Before writing code, generate the implementation spec for this task and present it to the FDE:
|
|
24
|
-
|
|
25
|
-
```
|
|
26
|
-
Spec: [task name from decisions.md]
|
|
27
|
-
Inputs: [what triggers this - event, request, user action, data shape]
|
|
28
|
-
Outputs: [what the user/system sees when it works]
|
|
29
|
-
Accepts:
|
|
30
|
-
- [scenario 1: specific testable outcome]
|
|
31
|
-
- [scenario 2: specific testable outcome]
|
|
32
|
-
Edge cases:
|
|
33
|
-
- [boundary condition]: [what happens]
|
|
34
|
-
- [error scenario]: [what happens]
|
|
35
|
-
Constraints: [performance, security, compliance bounds from trust-profile.md]
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
Present this to the FDE: "Here's what I'm about to build. Any open questions or changes before I start?"
|
|
39
|
-
|
|
40
|
-
- FDE says "yes" / "go" / "approved" → build against the spec exactly
|
|
41
|
-
- FDE modifies → update spec, confirm, then build
|
|
42
|
-
- Fast-track: if the task is under 30 minutes and the FDE has established trust (week 2+), state the spec inline and proceed unless they object
|
|
43
|
-
|
|
44
|
-
The spec becomes the test list - every line is something to verify after build.
|
|
45
|
-
|
|
46
|
-
## The loop (you do this work, in this order)
|
|
47
|
-
|
|
48
|
-
1. **Confirm scope in writing.** The task exists in `decisions.md` with acceptance criteria. Not there → plan first or name the scope creep.
|
|
49
|
-
2. **Declare blast radius.** Read the full dependency chain of what you're about to change. State: which files, which users, which systems. The change that breaks something always looked small.
|
|
50
|
-
3. **Confirm the rollback path exists** before writing a line. "We can always revert" is not a rollback path.
|
|
51
|
-
4. **Legacy code: characterisation tests first.** Tests for what the code *actually does right now* - including the parts that seem wrong; those behaviours are the contract the system depends on. Then wrap with Strangler Fig: new interface around old code, deprecate gradually, never rewrite in place.
|
|
52
|
-
5. **Search before creating.** Read the existing code in the area; don't add parallel helpers where a service exists. Integrating an SDK/vendor API → read real source (local `reference/repos/...` or official repo) before guessing names; record the files used in `decisions.md`. If an API looks invented, stop and search source.
|
|
53
|
-
6. **Build the minimal working path.** Thin vertical slice the customer can see. No opportunistic refactors. Every changed line traces to the task.
|
|
54
|
-
7. **Verify with evidence.** Run tests/typechecks/smallest proving script. State what ran and what didn't. "Seems right" is never evidence.
|
|
55
|
-
8. **Convergence check (spec vs. reality).** Walk through every acceptance scenario and edge case from the spec. State each one explicitly with its result:
|
|
56
|
-
- `[PASS]` scenario verified with evidence
|
|
57
|
-
- `[FAIL]` scenario not met - fix before proceeding
|
|
58
|
-
- `[DEFERRED]` intentionally left for a later task (state which one)
|
|
59
|
-
Surface the results to the FDE. If any scenario fails, fix it before cleanup. This is not optional - the spec is the contract.
|
|
60
|
-
9. **Cleanup pass after it works.** Dedupe repeated mechanics into the smallest service module; behavior unchanged; re-run the same tests. If you wrote 200 lines and 50 would do, rewrite before review.
|
|
61
|
-
10. **Review gate (before merge):** three beats, in order - (a) **session digest**: TL;DR + decisions/why + scope/verification into `.fde/` so the PR carries thinking, not just code (memory contract On exit; see `review.md`); (b) **intent vs diff**: every path KEEP / JUSTIFY / SPLIT / DROP against the stated slice in `decisions.md` (see `review.md` Stage 1); (c) **safety**: blast radius honest, tests meaningful, rollback real, secrets absent. Fix real findings, re-verify, repeat until clean or blocked on a human decision.
|
|
62
|
-
11. **Log and deliver.** Update the artifacts (below). Visible progress beats invisible perfection - every 2–3 tasks something shown to a stakeholder.
|
|
63
|
-
|
|
64
|
-
**Touching existing code - classify before changing:**
|
|
65
|
-
- **Fix now:** actively failing or blocking.
|
|
66
|
-
- **Fix when touched:** will bite when surrounding code changes - fix as part of this change.
|
|
67
|
-
- **Document and leave:** ugly but uninvolved. The hardest discipline. Opportunistic refactors create diffs nobody asked for and regressions nobody expected.
|
|
68
|
-
|
|
69
|
-
**AI components:** build the fallback path *before* the AI path (model slow? garbage? down?). Add observability before deploy: log inputs, outputs, confidence - hallucinations don't throw exceptions. Confirm the data processing agreement covers customer data before it goes to an external model; in regulated environments this is a blocker, not a detail.
|
|
70
|
-
|
|
71
|
-
**Regulated environments:** check the AI code policy in `trust-profile.md` first. Some modules require human review of AI-generated code or prohibit it. Find out before building, not after showing the code.
|
|
72
|
-
|
|
73
|
-
## Mid-build scope requests
|
|
74
|
-
|
|
75
|
-
"Also can you add…" → "That's worth looking at - let me place it." Current phase, future phase, or separate engagement? If it's outside `success.md`, name it: "Outside what we agreed; future phase or rescope conversation." Absorbed scope sets the precedent that the boundary doesn't exist - and in commercial engagements it silently moves billing and liability. Surface it to whoever owns the commercials.
|
|
76
|
-
|
|
77
|
-
## Method - integration design (connecting systems)
|
|
78
|
-
|
|
79
|
-
FDEs spend 40%+ of build time on integrations. Connecting system A to system B is where projects die quietly - wrong assumptions about APIs, missing auth, schema mismatches, rate limits.
|
|
80
|
-
|
|
81
|
-
**The integration checklist (before writing code):**
|
|
82
|
-
|
|
83
|
-
1. **Contract first.** Define what crosses the boundary: request shape, response shape, error shape. Write it down before calling anything. An undocumented contract changes without warning.
|
|
84
|
-
2. **Auth model.** OAuth2, API key, mTLS, SAML? Who provisions the credential? Who rotates it? What's the expiry? One expired token at 2am = production down.
|
|
85
|
-
3. **Rate limits and quotas.** Every external API has them, most aren't documented. Test with a burst early - don't discover the 429 on launch day.
|
|
86
|
-
4. **Error taxonomy.** Retryable (timeout, 503) vs. fatal (401, 422). Build retry with exponential backoff for the first, circuit-breaker for the second.
|
|
87
|
-
5. **Data transformation.** Their schema ≠ your schema. Map fields explicitly. Null handling rules. Timezone assumptions. Currency/locale. One unmapped field = silent data corruption.
|
|
88
|
-
6. **Idempotency.** Can you safely retry? If not, you need deduplication. Double-charges, double-posts, double-notifications - all integration bugs.
|
|
89
|
-
7. **Observability at the boundary.** Log every outbound call: request (redacted), response code, latency. The integration is the first thing to blame and the last thing instrumented.
|
|
90
|
-
|
|
91
|
-
**The integration contract template (write to `decisions.md`):**
|
|
92
|
-
```
|
|
93
|
-
Integration: [System A] → [System B]
|
|
94
|
-
Direction: push / pull / bidirectional
|
|
95
|
-
Auth: [method] - provisioned by [who] - expires [when]
|
|
96
|
-
Rate limit: [n] req/[period]
|
|
97
|
-
Retry: [strategy]
|
|
98
|
-
Data mapping: [field map or link to schema]
|
|
99
|
-
Failure mode: [what happens when B is down]
|
|
100
|
-
Owner: [who gets paged]
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
## Method - team amplification (making their engineers better)
|
|
104
|
-
|
|
105
|
-
The FDE's job is to make themselves replaceable. Not at handoff - every day. A client team that can only ship when the FDE is present is a dependency, not a success.
|
|
106
|
-
|
|
107
|
-
**Daily practices:**
|
|
108
|
-
- **Pair, don't take over.** When a client engineer is stuck, pair for 30 minutes rather than solving it alone in 10. The 10-minute fix creates dependency; the 30-minute pair creates capability.
|
|
109
|
-
- **Review their PRs with teaching intent.** Not "fix this" - "here's why this approach creates risk, and here's the pattern I'd use instead." Link to their own codebase for examples, not external docs.
|
|
110
|
-
- **Make decisions visible.** When you make an architecture choice, write a 3-line rationale in `decisions.md`. The team sees your reasoning, not just your output.
|
|
111
|
-
- **Name patterns, not just fixes.** "This is a circuit breaker pattern - here's when to use it again" converts one fix into a reusable skill.
|
|
112
|
-
|
|
113
|
-
**Weekly practices:**
|
|
114
|
-
- **One 30-min knowledge session per week.** Not a lecture - pick the hardest thing you built this week, walk through the thinking with the team. Answer: "why this approach and not the obvious one?"
|
|
115
|
-
- **Track the team's solo wins.** When a client engineer ships without FDE help, note it. This is the metric that proves the engagement is working.
|
|
116
|
-
|
|
117
|
-
**The independence signal:** When the team stops asking "how should we do X?" and starts asking "we're thinking X because Y - does that sound right?" - the engagement is succeeding.
|
|
118
|
-
|
|
119
|
-
## Stop signals - reassess immediately
|
|
120
|
-
|
|
121
|
-
- "I'll add tests after this works" - the regression is already in.
|
|
122
|
-
- "Small change, no need to declare blast radius" - famous last words.
|
|
123
|
-
- "I'll refactor this while I'm here" - unasked-for risk in an unrelated change.
|
|
124
|
-
- **Three fixes in and still broken - the diagnosis is wrong.** No fourth fix: state your current model of the problem, name the evidence that would disprove it, test that first (see `debug.md`).
|
|
125
|
-
|
|
126
|
-
## Artifact (logged as you go, not after)
|
|
127
|
-
|
|
128
|
-
- **`decisions.md`** - each significant choice: what, alternatives considered, why this one. For non-trivial architecture decisions, present three options to the FDE (safe / pragmatic / aggressive) with costs and a recommendation - three options is a real decision; one option is a request for trust. Integration contracts go here too.
|
|
129
|
-
- **`risks.md`** - new risks discovered while building.
|
|
130
|
-
- **`delivery.md`** - append a **value ledger** row for every ship: Date | Slice | Bucket | Promised | Measured | Accepted by | Evidence | Rollback. Bucket is `cost-save` / `risk-mitigation` / `revenue-uplift`. "Measured" may be `pending` until the pulse exists - never skip the promised column. Narrative under Shipped is optional color; the ledger is the record status and close read.
|
|
131
|
-
|
|
132
|
-
**Measured is not the same as accepted.** A number the FDE calculated is `claimed` until a named customer-side owner agrees it is real - their finance lead, their ops manager, the person whose budget it moves. Write the name and the date in **Accepted by**; leave it empty and the row reads `claimed`, never "delivered". This is the difference between a benefit that survives the renewal conversation and one the sponsor's CFO deletes in the review.
|
|
133
|
-
|
|
134
|
-
## Checkpoint
|
|
135
|
-
|
|
136
|
-
Before merge: the two-stage review (scope, then safety) has run clean, verification evidence is stated, and the slice is demonstrable. If `trust-profile.md` requires human sign-off on AI-generated code, that sign-off exists.
|
|
137
|
-
|
|
138
|
-
## Worked example
|
|
139
|
-
|
|
140
|
-
Acme task 1: route reconciliation failures to a named on-call.
|
|
141
|
-
|
|
142
|
-
Characterisation first — a test that captures what the job does *today* when it exits non-zero (silently succeeds from the caller's perspective). That test fails after the fix, which is the point: it documents the behaviour being changed rather than trusting the diff.
|
|
143
|
-
|
|
144
|
-
Mid-build, Tom asks to "just also fix the retry logic while you're in there." That is a scope decision, not a request: logged in `decisions.md` with the blast radius (retry touches the settlement path — not in this slice's declared radius) and routed to the plan's Next lane.
|
|
145
|
-
|
|
146
|
-
Ledger row on ship: `Jun 18 | failure routing | risk-mitigation | 4h → 15min detection | pending | — | staging kill test, PR #212 | disable alert route`. **Accepted by** stays empty until Marco confirms from a real incident — until then the number is claimed, and the status update says so.
|
|
147
|
-
|
|
148
|
-
## Principles
|
|
149
|
-
|
|
150
|
-
- Characterisation tests before modification. Every time.
|
|
151
|
-
- Blast radius declared before every change; rollback confirmed before every deploy.
|
|
152
|
-
- Scope creep is a decision, not a request. Name it.
|
|
153
|
-
- Build the fallback before the AI feature.
|
|
154
|
-
- Small diffs, objective verification, no thousand-line hope PRs.
|
|
@@ -1,55 +0,0 @@
|
|
|
1
|
-
# debug - systematic, never guessing
|
|
2
|
-
|
|
3
|
-
**Enter when:** something is broken, can't be reproduced, or "shouldn't be happening" - and production is NOT down (that's `rescue.md`).
|
|
4
|
-
|
|
5
|
-
**Read first:** `context.md`, `terrain.md`, `chaos-log.md` (if this happened before, the cause is probably the same). Nothing else - noise kills focus.
|
|
6
|
-
|
|
7
|
-
Debugging on an engagement differs from debugging your own code: no full context, no history, high pressure to "just fix it." That pressure causes the second incident. The sequence below is the protection.
|
|
8
|
-
|
|
9
|
-
## Method (you do this work - never skip a step)
|
|
10
|
-
|
|
11
|
-
**1. Reproduce.** Get a consistent repro before looking at any code. Can't reproduce → instrument first (logging, narrowing) - not fixes. An unreproducible bug that gets "fixed" comes back.
|
|
12
|
-
|
|
13
|
-
**2. Isolate to the smallest failing case.** Strip everything that isn't the failure. Payment flow bug → does one hard-coded test transaction fail? Smallest case = smallest context = smallest blast radius for the fix.
|
|
14
|
-
|
|
15
|
-
**3. Ask "what changed?" before reading code.** Last 2 hours, 24 hours, last deploy - this one question solves most production bugs:
|
|
16
|
-
```bash
|
|
17
|
-
git log --since="48 hours ago" --format="%ad %an %s" --date=relative
|
|
18
|
-
git log --stat -5 --format="%h %s" # works on any history depth
|
|
19
|
-
```
|
|
20
|
-
Don't read the codebase like a book. Write a script that answers one specific question - one execution, one answer.
|
|
21
|
-
|
|
22
|
-
**4. One hypothesis.** Not three. State it explicitly:
|
|
23
|
-
> "Hypothesis: X. If right, changing Y fixes it. If wrong, the symptom persists."
|
|
24
|
-
A hypothesis that can't be falsified isn't one.
|
|
25
|
-
|
|
26
|
-
**5. Fix the root cause, not the symptom.** A 500 is a symptom; an NPE is a symptom; the cause is upstream. Patch the symptom and you're back in a week.
|
|
27
|
-
|
|
28
|
-
**6. Verify the fix holds.** Repro case passes; tests covering the changed code + downstream callers green; no new errors in logs for N minutes. Define done *before* declaring it - "seems fixed" requires another cycle.
|
|
29
|
-
|
|
30
|
-
## Talking to the customer mid-debug (you coach)
|
|
31
|
-
|
|
32
|
-
They'll ask for status before there is one. The FDE gives: what's narrowed, what's ruled out, what's being tested next, and **a specific time** they'll hear back. Proactive update if past that time - never make them chase. Skip the phrases that add heat: "weird one," "never seen this," "might be…". Partial clarity beats raw uncertainty. Fixed without root cause → stabilise honestly, don't close the story until the why is explainable.
|
|
33
|
-
|
|
34
|
-
## Artifact
|
|
35
|
-
|
|
36
|
-
**`chaos-log.md`** - append, same day:
|
|
37
|
-
```markdown
|
|
38
|
-
## <date> - <symptom>
|
|
39
|
-
**Root cause:** <not the symptom>
|
|
40
|
-
**What changed to fix it:** <change>
|
|
41
|
-
**Hypotheses tested:** <in order, results>
|
|
42
|
-
**Recurrence risk:** <where this can happen again>
|
|
43
|
-
```
|
|
44
|
-
Update `terrain.md` if the investigation disproved something the map claimed.
|
|
45
|
-
|
|
46
|
-
## Checkpoint
|
|
47
|
-
|
|
48
|
-
Before closing: repro passes, root cause stated in one sentence, chaos log written. If the root cause is still unknown, the incident stays open - say so.
|
|
49
|
-
|
|
50
|
-
## Principles
|
|
51
|
-
|
|
52
|
-
- Reproduce before touching anything. Always.
|
|
53
|
-
- "What changed?" is the first question, not the last.
|
|
54
|
-
- One hypothesis at a time. Three fixes failed = wrong mental model - stop.
|
|
55
|
-
- Fix upstream. Symptoms patched are incidents scheduled.
|
|
@@ -1,103 +0,0 @@
|
|
|
1
|
-
# observability - if you can't see it running, you can't operate it
|
|
2
|
-
|
|
3
|
-
**Enter when:** shipping a feature to production, the customer says "we don't know when things break," a post-incident review revealed gaps in monitoring, or the handoff needs the team to operate what was built.
|
|
4
|
-
|
|
5
|
-
**Read first:** `terrain.md`, `delivery.md`, `context.md`. Load `trust-profile.md` if the data flowing through logs contains sensitive information.
|
|
6
|
-
|
|
7
|
-
Code without observability is code you can't operate. The FDE who ships a feature without telemetry creates a callback in six weeks when it breaks and nobody can tell why. Observability is built alongside the feature, not after - the same way tests are.
|
|
8
|
-
|
|
9
|
-
## Method (you do this work)
|
|
10
|
-
|
|
11
|
-
**1. Define "working" before instrumenting.** Write 2–4 questions that the person on call will ask about this feature:
|
|
12
|
-
|
|
13
|
-
```
|
|
14
|
-
FEATURE: payment retry logic
|
|
15
|
-
QUESTIONS ON-CALL WILL ASK:
|
|
16
|
-
1. What fraction of payments succeed on first attempt vs after retry?
|
|
17
|
-
2. When a payment fails permanently, why? (provider error? timeout? validation?)
|
|
18
|
-
3. Is the payment provider slower than usual?
|
|
19
|
-
4. Are retries causing duplicate charges?
|
|
20
|
-
→ Every signal below must help answer one of these.
|
|
21
|
-
```
|
|
22
|
-
|
|
23
|
-
If you can't name the questions, you're not ready to instrument - you'll log everything and learn nothing.
|
|
24
|
-
|
|
25
|
-
**2. Pick the right signal for each question:**
|
|
26
|
-
|
|
27
|
-
| Signal | Answers | When to use |
|
|
28
|
-
|--------|---------|-------------|
|
|
29
|
-
| **Structured log** | "What happened in this specific case?" | Individual request debugging, audit trails |
|
|
30
|
-
| **Metric** | "How often / how fast, in aggregate?" | Dashboards, alerting, trend detection |
|
|
31
|
-
| **Trace** | "Where did time go across services?" | Cross-service latency, bottleneck identification |
|
|
32
|
-
|
|
33
|
-
Rule of thumb: metrics tell you **that** something is wrong, traces tell you **where**, logs tell you **why**.
|
|
34
|
-
|
|
35
|
-
**3. Structured logging - events, not prose.**
|
|
36
|
-
|
|
37
|
-
```
|
|
38
|
-
BAD: logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`)
|
|
39
|
-
→ Unqueryable, inconsistent format, buried in noise
|
|
40
|
-
|
|
41
|
-
GOOD: logger.info({ event: "payment_failed", payment_id: id, user_id: userId,
|
|
42
|
-
retry_count: n, error_code: "PROVIDER_TIMEOUT", duration_ms: elapsed })
|
|
43
|
-
→ Queryable, alertable, structured for dashboards
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
Every log line: a stable event name + machine-readable fields. Human-readable messages are for debugging sessions; structured events are for production operations.
|
|
47
|
-
|
|
48
|
-
**4. The four metrics every FDE feature needs:**
|
|
49
|
-
|
|
50
|
-
| Metric | What it measures | Alert threshold |
|
|
51
|
-
|--------|-----------------|-----------------|
|
|
52
|
-
| **Error rate** | Failures / total requests | >2x baseline for 5 minutes |
|
|
53
|
-
| **Latency (p95/p99)** | How long the operation takes | >2x normal for 5 minutes |
|
|
54
|
-
| **Throughput** | Requests per second/minute | <50% of normal for 10 minutes (demand drop = upstream problem) |
|
|
55
|
-
| **Business metric** | The thing the feature is supposed to improve | Direction reverses for 1 hour |
|
|
56
|
-
|
|
57
|
-
The business metric is the one most FDEs skip and the one the sponsor cares about most. "Error rate is fine" means nothing if payments processed per hour dropped.
|
|
58
|
-
|
|
59
|
-
**5. Alert design - avoid noise, ensure action:**
|
|
60
|
-
|
|
61
|
-
| Principle | How |
|
|
62
|
-
|-----------|-----|
|
|
63
|
-
| **Every alert has an owner** | If nobody is named, nobody responds |
|
|
64
|
-
| **Every alert has a runbook** | "Error rate spiked" → "Check these three things in this order" |
|
|
65
|
-
| **Severity maps to response time** | Critical: 15 min. Warning: next business day. Info: weekly review. |
|
|
66
|
-
| **No alert without action** | If the response is "ignore it" three times, delete the alert |
|
|
67
|
-
|
|
68
|
-
**6. AI component observability.** AI features fail differently - they degrade, they don't crash:
|
|
69
|
-
|
|
70
|
-
| What to observe | Why |
|
|
71
|
-
|----------------|-----|
|
|
72
|
-
| Model input/output pairs (sampled, privacy-respecting) | Detect drift: outputs changing without code changing |
|
|
73
|
-
| Latency per model call | Provider degradation is gradual, not binary |
|
|
74
|
-
| Fallback activation rate | If fallback fires >5%, the primary path has a problem |
|
|
75
|
-
| Confidence/quality score trend | A score that slowly drops = model drift |
|
|
76
|
-
| Token usage / cost per request | Cost creep is invisible until the bill arrives |
|
|
77
|
-
|
|
78
|
-
**7. The sacred-data boundary.** Before shipping any observability:
|
|
79
|
-
|
|
80
|
-
- Check `trust-profile.md` for `<private>` tagged data.
|
|
81
|
-
- PII in logs = a breach, not a debug aid. Mask, hash, or exclude.
|
|
82
|
-
- In healthcare: PHI in logs violates HIPAA. In fintech: PANs in logs violate PCI-DSS.
|
|
83
|
-
- Default: log IDs, not values. `user_id: 12345` not `user_email: jane@...`
|
|
84
|
-
|
|
85
|
-
## Artifact
|
|
86
|
-
|
|
87
|
-
**`delivery.md`** - under each shipped feature: what's observable, where to look, alert thresholds, the runbook.
|
|
88
|
-
|
|
89
|
-
**`handoff.md`** (when close approaches) - the on-call guide: "what each alert means and what to do."
|
|
90
|
-
|
|
91
|
-
**`risks.md`** - if observability gaps were found in existing code: what's unobserved and the risk it carries.
|
|
92
|
-
|
|
93
|
-
## Checkpoint
|
|
94
|
-
|
|
95
|
-
Before marking a feature shipped: the four metrics are emitting, alerts have owners and runbooks, sacred data is not in logs. If any gap: "Observability incomplete - <specific gap> - shipping without it means <specific risk>."
|
|
96
|
-
|
|
97
|
-
## Principles
|
|
98
|
-
|
|
99
|
-
- Define the questions before choosing the signals. Unquestioned telemetry is noise.
|
|
100
|
-
- Metrics for detection, traces for location, logs for explanation.
|
|
101
|
-
- Every alert has an owner and a runbook, or it's noise.
|
|
102
|
-
- AI features need observability for drift, not just failures.
|
|
103
|
-
- PII in logs is a breach. Default to IDs, not values.
|
|
@@ -1,113 +0,0 @@
|
|
|
1
|
-
# qa-live - test it like a user, not like an engineer
|
|
2
|
-
|
|
3
|
-
**Enter when:** a feature is built and needs to be verified from the user's perspective, the team says "it works on my machine," a demo is coming and the feature hasn't been clicked through, or a post-deploy smoke test is needed.
|
|
4
|
-
|
|
5
|
-
**Read first:** `delivery.md` (what was built), `decisions.md` (acceptance criteria), `success.md` (what the customer expects to see), `context.md`.
|
|
6
|
-
|
|
7
|
-
Unit tests prove the code works. QA proves the *feature* works - from the user's chair, on a real browser, with real data patterns. The FDE who ships a feature that passes all tests but fails the first user click has shipped a defect.
|
|
8
|
-
|
|
9
|
-
## Method (you do this work)
|
|
10
|
-
|
|
11
|
-
**1. Write the test script from the user's story, not the code.**
|
|
12
|
-
|
|
13
|
-
Start with `success.md` and `decisions.md` acceptance criteria. For each acceptance criterion, write the human steps:
|
|
14
|
-
|
|
15
|
-
```markdown
|
|
16
|
-
## Test: User can retry a failed payment
|
|
17
|
-
Precondition: User has a failed payment in their history
|
|
18
|
-
Steps:
|
|
19
|
-
1. Navigate to payment history
|
|
20
|
-
2. Click the failed payment
|
|
21
|
-
3. Click "Retry payment"
|
|
22
|
-
4. Confirm the retry dialog
|
|
23
|
-
5. Observe the result
|
|
24
|
-
Expected: Payment processes successfully, status updates to "Completed"
|
|
25
|
-
Also check: Error message if retry fails, loading state during processing
|
|
26
|
-
```
|
|
27
|
-
|
|
28
|
-
**2. The five perspectives.** Test each feature from five angles that unit tests can't reach:
|
|
29
|
-
|
|
30
|
-
| Perspective | What to test | Common FDE finding |
|
|
31
|
-
|------------|-------------|-------------------|
|
|
32
|
-
| **Happy path** | Does the main flow work end-to-end? | Works locally, fails with real data volumes |
|
|
33
|
-
| **Error path** | What happens when things go wrong? | Error messages are developer-facing, not user-facing |
|
|
34
|
-
| **Edge cases** | Empty states, max lengths, special characters, concurrent users | The empty state shows "undefined" instead of a helpful message |
|
|
35
|
-
| **Performance** | Is it fast enough for real use? | Works fine with 10 records, unusable with 10,000 |
|
|
36
|
-
| **Accessibility** | Can it be used with keyboard only? Does the screen reader make sense? | Tab order is broken, focus traps in modals |
|
|
37
|
-
|
|
38
|
-
**3. Real browser testing.** Open the actual application in a browser and click through:
|
|
39
|
-
|
|
40
|
-
```
|
|
41
|
-
Before each test:
|
|
42
|
-
- Clear relevant caches/state
|
|
43
|
-
- Use realistic test data (not "test test test")
|
|
44
|
-
- Note the browser and viewport size
|
|
45
|
-
|
|
46
|
-
During each test:
|
|
47
|
-
- Open DevTools console - watch for errors
|
|
48
|
-
- Open DevTools network tab - watch for failed requests
|
|
49
|
-
- Time the critical actions (user-perceivable latency)
|
|
50
|
-
|
|
51
|
-
After each test:
|
|
52
|
-
- Screenshot the result (before/after when relevant)
|
|
53
|
-
- Note any console errors even if the test "passed"
|
|
54
|
-
- Note any UX friction even if technically correct
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
**4. The health score.** Rate the feature across five dimensions, 1–5:
|
|
58
|
-
|
|
59
|
-
| Dimension | 1 (broken) | 3 (acceptable) | 5 (polished) |
|
|
60
|
-
|-----------|-----------|----------------|--------------|
|
|
61
|
-
| **Functionality** | Critical path fails | Happy path works, errors unhandled | All paths work, errors graceful |
|
|
62
|
-
| **Performance** | >5s load time | <2s load time | <500ms load time, no layout shifts |
|
|
63
|
-
| **Error handling** | Crashes or shows stack trace | Shows error message | Shows actionable error with recovery option |
|
|
64
|
-
| **Data handling** | Corrupts or loses data | Handles normal data correctly | Handles edge cases (empty, large, special chars) |
|
|
65
|
-
| **User experience** | Confusing or broken layout | Functional but rough | Intuitive, consistent with the rest of the app |
|
|
66
|
-
|
|
67
|
-
**Overall health = average of five scores.** Below 3.0 → not ready to ship. 3.0–4.0 → shippable with known issues. 4.0+ → confident to demo.
|
|
68
|
-
|
|
69
|
-
**5. The bug report format.** For each issue found:
|
|
70
|
-
|
|
71
|
-
```markdown
|
|
72
|
-
## Bug: <one-line summary>
|
|
73
|
-
Severity: critical / high / medium / low
|
|
74
|
-
Steps to reproduce:
|
|
75
|
-
1. <exact steps>
|
|
76
|
-
2. <exact steps>
|
|
77
|
-
Expected: <what should happen>
|
|
78
|
-
Actual: <what happened>
|
|
79
|
-
Evidence: <screenshot, console error, network request>
|
|
80
|
-
Environment: <browser, viewport, data state>
|
|
81
|
-
```
|
|
82
|
-
|
|
83
|
-
**6. Re-verify after fixes.** After each bug is fixed:
|
|
84
|
-
- Re-run the exact reproduction steps
|
|
85
|
-
- Check that the fix didn't break adjacent functionality
|
|
86
|
-
- Update the health score
|
|
87
|
-
- Screenshot the fixed state
|
|
88
|
-
|
|
89
|
-
## Artifact
|
|
90
|
-
|
|
91
|
-
**`delivery.md`** - the health score and test results for each shipped feature:
|
|
92
|
-
```markdown
|
|
93
|
-
## QA: <feature name> - <date>
|
|
94
|
-
Health score: 4.2 / 5.0
|
|
95
|
-
Tests run: 8 passed, 1 failed (fixed), 1 known issue (low severity)
|
|
96
|
-
Bugs found: 2 (1 fixed, 1 deferred to next sprint)
|
|
97
|
-
Ready to demo: YES / NO
|
|
98
|
-
```
|
|
99
|
-
|
|
100
|
-
**`decisions.md`** - bugs deferred with rationale (why it's acceptable to ship with this known issue).
|
|
101
|
-
|
|
102
|
-
## Checkpoint
|
|
103
|
-
|
|
104
|
-
One line: "Feature tested from user perspective. Health score: <N>/5. <N> bugs found, <N> fixed, <N> deferred. Ready to demo: yes/no." If not ready: the specific blocker.
|
|
105
|
-
|
|
106
|
-
## Principles
|
|
107
|
-
|
|
108
|
-
- Test from the user's chair, not the developer's IDE.
|
|
109
|
-
- Real browser, real data patterns, real network conditions.
|
|
110
|
-
- Console errors during a "passing" test are still findings.
|
|
111
|
-
- Health score below 3.0 = not ready to ship, regardless of test suite.
|
|
112
|
-
- Every bug gets a reproduction recipe, not a description.
|
|
113
|
-
- A feature that passes all unit tests but fails the first click is a defect.
|
|
@@ -1,105 +0,0 @@
|
|
|
1
|
-
# security-audit - find the holes before someone else does
|
|
2
|
-
|
|
3
|
-
**Enter when:** the engagement touches auth, payments, user data, or external integrations; the customer mentions compliance (SOC2, PCI, HIPAA); a new API endpoint is being shipped; or the FDE is asked "is this secure?"
|
|
4
|
-
|
|
5
|
-
**Read first:** `trust-profile.md` (data classification, AI policy, compliance requirements), `terrain.md`, `context.md`. Load the relevant regulated overlay (fintech/healthcare/gov) if the signal is present.
|
|
6
|
-
|
|
7
|
-
On an FDE engagement, security mistakes are twice as dangerous: you break someone else's system, with their users' data, under their compliance obligations. The FDE who finds the vulnerability before production earns trust that lasts the entire engagement.
|
|
8
|
-
|
|
9
|
-
## Method (you do this work)
|
|
10
|
-
|
|
11
|
-
**1. Threat model in five minutes.** Not a ceremony - five questions before looking at code:
|
|
12
|
-
|
|
13
|
-
| Question | What it reveals |
|
|
14
|
-
|----------|----------------|
|
|
15
|
-
| Where does untrusted data enter? | HTTP requests, file uploads, webhooks, LLM outputs, message queues |
|
|
16
|
-
| What's worth stealing? | Credentials, PII, payment data, API keys, session tokens |
|
|
17
|
-
| What's worth breaking? | Auth system, payment flow, admin actions, data integrity |
|
|
18
|
-
| Who's the attacker? | External (internet), internal (employee), adjacent (other tenant), automated (bot) |
|
|
19
|
-
| What's the worst realistic outcome? | Data breach, financial loss, regulatory fine, reputation damage |
|
|
20
|
-
|
|
21
|
-
Write the answers before scanning code. The threat model tells you where to look; code scanning without a threat model is reading every room in a building instead of checking the doors.
|
|
22
|
-
|
|
23
|
-
**2. The STRIDE pass.** For each trust boundary (where data crosses from untrusted to trusted):
|
|
24
|
-
|
|
25
|
-
| Threat | Check | Common FDE finding |
|
|
26
|
-
|--------|-------|-------------------|
|
|
27
|
-
| **Spoofing** | Can someone impersonate a user/service? | Missing auth on internal endpoints ("it's behind the VPN" is not auth) |
|
|
28
|
-
| **Tampering** | Can data be altered in transit or at rest? | Unparameterised SQL, unsigned webhooks, client-side validation as sole gate |
|
|
29
|
-
| **Repudiation** | Can an action be denied later? | No audit log on admin actions, no timestamp on state changes |
|
|
30
|
-
| **Information disclosure** | Can data leak? | Stack traces in production errors, PII in logs, verbose error messages |
|
|
31
|
-
| **Denial of service** | Can it be overwhelmed? | No rate limiting on auth endpoints, unbounded file uploads, no pagination |
|
|
32
|
-
| **Elevation of privilege** | Can a user gain access they shouldn't? | IDOR (changing user ID in URL), missing role checks on endpoints |
|
|
33
|
-
|
|
34
|
-
**3. The automated scan.** Run these - they catch what manual review misses:
|
|
35
|
-
|
|
36
|
-
```bash
|
|
37
|
-
# Secrets in code (repeat --include per extension)
|
|
38
|
-
grep -rnE "(api[_-]?key|secret|password|token|private[_-]?key)\s*[:=]\s*['\"][^'\"]{8,}" \
|
|
39
|
-
--include="*.js" --include="*.ts" --include="*.py" --include="*.env" \
|
|
40
|
-
--include="*.yaml" --include="*.json" . | grep -vE "example|template|test|mock" | head -20
|
|
41
|
-
|
|
42
|
-
# Dependency vulnerabilities
|
|
43
|
-
npm audit 2>/dev/null || pip audit 2>/dev/null || echo "no package audit available"
|
|
44
|
-
|
|
45
|
-
# SQL injection patterns
|
|
46
|
-
grep -rnE "(SELECT|INSERT|UPDATE|DELETE).*\+.*\"|f['\"].*{.*}.*SELECT" \
|
|
47
|
-
--include="*.js" --include="*.ts" --include="*.py" . | head -10
|
|
48
|
-
|
|
49
|
-
# Dangerous patterns
|
|
50
|
-
grep -rnE "eval\(|innerHTML\s*=|document\.write\(|exec\(|__import__" \
|
|
51
|
-
--include="*.js" --include="*.ts" --include="*.py" . | head -10
|
|
52
|
-
```
|
|
53
|
-
|
|
54
|
-
**4. The AI security check.** If the system uses AI/LLM components:
|
|
55
|
-
|
|
56
|
-
| Check | Finding if yes |
|
|
57
|
-
|-------|---------------|
|
|
58
|
-
| Is model output used in SQL, shell commands, or HTML without sanitisation? | Injection via prompt - treat model output as untrusted input |
|
|
59
|
-
| Is the system prompt relied on as a security boundary? | Prompt injection bypasses it - enforce permissions in code |
|
|
60
|
-
| Are secrets or cross-tenant data in the context window? | Data leakage via prompt extraction |
|
|
61
|
-
| Are tool/agent permissions scoped? | Excessive agency - model can take actions it shouldn't |
|
|
62
|
-
| Are token/rate/recursion limits set? | Unbounded consumption or infinite loops |
|
|
63
|
-
|
|
64
|
-
**5. Classify findings by severity and action:**
|
|
65
|
-
|
|
66
|
-
| Severity | Criteria | Action |
|
|
67
|
-
|----------|----------|--------|
|
|
68
|
-
| **Critical** | Exploitable now, real data at risk | Stop other work. Fix before next merge. |
|
|
69
|
-
| **High** | Exploitable with effort, or compliance violation | Fix this phase. Track in `risks.md`. |
|
|
70
|
-
| **Medium** | Defense-in-depth gap, no immediate exploit | Log it. Fix when the module is next touched. |
|
|
71
|
-
| **Low** | Best practice gap, no exploit path | Note for the handoff document. |
|
|
72
|
-
|
|
73
|
-
**6. Present findings as protection, not criticism.**
|
|
74
|
-
|
|
75
|
-
The internal team built this system under constraints. Frame findings as shared wins:
|
|
76
|
-
|
|
77
|
-
> "Found three endpoints without rate limiting - adding that now prevents the bot attack that hit [company in their industry] last quarter. Here's the fix."
|
|
78
|
-
|
|
79
|
-
Not: "Your auth is broken." Even if it is.
|
|
80
|
-
|
|
81
|
-
## Artifact
|
|
82
|
-
|
|
83
|
-
**`risks.md`** - each finding with severity, evidence, remediation, and status:
|
|
84
|
-
```markdown
|
|
85
|
-
## Security finding: <title> - <severity>
|
|
86
|
-
Where: <file:line or endpoint>
|
|
87
|
-
Evidence: <what you found>
|
|
88
|
-
Risk: <what an attacker could do>
|
|
89
|
-
Remediation: <specific fix>
|
|
90
|
-
Status: fixed / in-progress / logged-for-handoff
|
|
91
|
-
```
|
|
92
|
-
|
|
93
|
-
**`trust-profile.md`** - update if new sensitive data paths, AI components, or compliance requirements were discovered.
|
|
94
|
-
|
|
95
|
-
## Checkpoint
|
|
96
|
-
|
|
97
|
-
Summary to the FDE: findings by severity count, the one critical/high finding that needs immediate attention (if any), and the overall security posture in one sentence. If no critical findings: "No exploitable issues found in this pass - next audit at <trigger>."
|
|
98
|
-
|
|
99
|
-
## Principles
|
|
100
|
-
|
|
101
|
-
- Threat model first, scan second. Know where to look before looking everywhere.
|
|
102
|
-
- Treat model output as untrusted input - always.
|
|
103
|
-
- Frame findings as protection, not criticism. The team built under constraints.
|
|
104
|
-
- Secrets in code outrank every other finding. Check first.
|
|
105
|
-
- A security audit with no findings either found nothing or didn't look hard enough - state which.
|
|
@@ -1,108 +0,0 @@
|
|
|
1
|
-
# test-on-legacy - making changes safe on code that has no tests
|
|
2
|
-
|
|
3
|
-
**Enter when:** the codebase has little or no test coverage, you need to change code where tests are absent or misleading, or `terrain.md` flagged high-churn modules with no test neighbours.
|
|
4
|
-
|
|
5
|
-
**Read first:** `terrain.md` (the churn heat map), `decisions.md` (the current slice), `context.md`. This skill is the safety net for building on someone else's untested codebase.
|
|
6
|
-
|
|
7
|
-
Legacy code without tests is a minefield. You can't refactor it because you don't know what it does. You can't add features because you don't know what you'll break. The way through: characterise what exists, wrap the change, prove it works - in that order.
|
|
8
|
-
|
|
9
|
-
## Method (you do this work)
|
|
10
|
-
|
|
11
|
-
**1. Characterisation tests first.** Before changing anything, write tests that describe what the code *actually does right now* - including the parts that seem wrong:
|
|
12
|
-
|
|
13
|
-
```
|
|
14
|
-
The code truncates names at 50 characters.
|
|
15
|
-
→ That seems like a bug, but it might be a contract another system depends on.
|
|
16
|
-
→ Write a test: "truncates names at 50 characters" - that's the characterisation.
|
|
17
|
-
→ NOW you can change the code and know exactly what you've broken.
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
Characterisation tests answer: "What does this code do?" not "What should this code do?" They're the honest documentation that the README isn't.
|
|
21
|
-
|
|
22
|
-
**How to write them:**
|
|
23
|
-
1. Pick the function/module you're about to change.
|
|
24
|
-
2. Call it with representative inputs (from production if possible, from logs, from the team's knowledge).
|
|
25
|
-
3. Record what comes back - that's your expected output.
|
|
26
|
-
4. Turn that into an assertion.
|
|
27
|
-
|
|
28
|
-
```
|
|
29
|
-
# The pattern:
|
|
30
|
-
result = function_under_test(real_input)
|
|
31
|
-
assert result == whatever_it_actually_returned # characterisation, not specification
|
|
32
|
-
```
|
|
33
|
-
|
|
34
|
-
**2. The Strangler Fig pattern - wrap, don't rewrite.**
|
|
35
|
-
|
|
36
|
-
Never rewrite legacy code in place. Instead:
|
|
37
|
-
|
|
38
|
-
```
|
|
39
|
-
Step 1: New interface wraps the old code (calls through to it)
|
|
40
|
-
→ All existing callers work exactly as before
|
|
41
|
-
→ Your characterisation tests pass
|
|
42
|
-
|
|
43
|
-
Step 2: New implementation behind the new interface
|
|
44
|
-
→ Old code still there, still callable
|
|
45
|
-
→ Feature flag or config switches between old and new
|
|
46
|
-
|
|
47
|
-
Step 3: Gradually migrate callers to the new path
|
|
48
|
-
→ Each migration is a small, testable change
|
|
49
|
-
→ Old path remains as fallback
|
|
50
|
-
|
|
51
|
-
Step 4: Remove old code only when:
|
|
52
|
-
→ No callers remain
|
|
53
|
-
→ New path has been stable for N days
|
|
54
|
-
→ Team agrees it's safe
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
**3. The test pyramid for legacy engagement work:**
|
|
58
|
-
|
|
59
|
-
| Level | What to write | How many | Why |
|
|
60
|
-
|-------|-------------|---------|-----|
|
|
61
|
-
| **Characterisation** | What the code does now | 3–5 per module you're changing | Safety net before any change |
|
|
62
|
-
| **Unit** | Your new code's behaviour | 1 per new function/method | Proves your addition works |
|
|
63
|
-
| **Integration** | The seam between old and new | 1–2 per boundary | Proves old and new cooperate |
|
|
64
|
-
| **Smoke** | The critical user path end-to-end | 1 per feature | Proves the user can still do the thing |
|
|
65
|
-
|
|
66
|
-
**4. Spot the lying tests.** Worse than no tests are tests that pass but verify nothing:
|
|
67
|
-
|
|
68
|
-
```
|
|
69
|
-
# This test passes and proves nothing:
|
|
70
|
-
def test_process_payment():
|
|
71
|
-
result = process_payment(mock_everything())
|
|
72
|
-
assert result is not None # what does "not None" prove?
|
|
73
|
-
|
|
74
|
-
# This test is actually testing something:
|
|
75
|
-
def test_process_payment_deducts_from_balance():
|
|
76
|
-
account = create_account(balance=100)
|
|
77
|
-
process_payment(account, amount=30)
|
|
78
|
-
assert account.balance == 70
|
|
79
|
-
```
|
|
80
|
-
|
|
81
|
-
When you find a lying test: note it in `terrain.md`. Don't fix it unless it's in your slice - but name it, because the next person needs to know.
|
|
82
|
-
|
|
83
|
-
**5. The "safe to change" checklist.** Before modifying any legacy code:
|
|
84
|
-
|
|
85
|
-
- [ ] Characterisation tests written for the module being changed
|
|
86
|
-
- [ ] All characterisation tests pass before your change
|
|
87
|
-
- [ ] Your change is wrapped (Strangler Fig), not a rewrite-in-place
|
|
88
|
-
- [ ] New tests cover your new behaviour
|
|
89
|
-
- [ ] All tests (characterisation + new) pass after your change
|
|
90
|
-
- [ ] The diff shows only what you intended to change
|
|
91
|
-
|
|
92
|
-
## Artifact
|
|
93
|
-
|
|
94
|
-
**`terrain.md`** - update test-gap assessment: which modules now have characterisation tests, which still don't, which tests are lying.
|
|
95
|
-
|
|
96
|
-
**`decisions.md`** - log: "Added characterisation tests for <module> before changing <feature>. Coverage state: <before/after>."
|
|
97
|
-
|
|
98
|
-
## Checkpoint
|
|
99
|
-
|
|
100
|
-
Before merging any change to legacy code: characterisation tests existed before the change (state which), new tests cover the new behaviour, all pass. If characterisation tests were skipped: that's a finding - state why (time pressure? inaccessible code?) and log the risk.
|
|
101
|
-
|
|
102
|
-
## Principles
|
|
103
|
-
|
|
104
|
-
- Characterise before changing. What the code does > what it should do.
|
|
105
|
-
- Wrap, don't rewrite. The Strangler Fig is the safest pattern on legacy code.
|
|
106
|
-
- A lying test is worse than no test. Name it when you find it.
|
|
107
|
-
- The ugly behaviour in the characterisation test might be someone else's contract. Don't "fix" it without asking.
|
|
108
|
-
- Test coverage on legacy code is insurance - buy it before you need it, not after.
|