@mmerterden/multi-agent-pipeline 20.2.1 → 20.3.1

package/pipeline/skills/shared/external/debugging-strategies/SKILL.md

@@ -1,33 +1,143 @@
 ---
 name: debugging-strategies
-description: "Transform debugging from frustrating guesswork into systematic problem-solving with proven strategies, powerful tools, and methodical approaches. Use when a bug resists guesswork and needs a systematic approach."
-risk: safe
-source: community
-date_added: "2026-02-27"
+description: "Language-neutral debugging method: reproduce, minimise, falsifiable hypotheses, git bisect, instrumentation, production traces, concurrency bugs, regression test. Use when a bug resists guesswork."
+metadata:
+  risk: safe
+  source: multi-agent-pipeline
+  date_added: "2026-09-25"
 ---
 
 # Debugging Strategies
 
-Transform debugging from frustrating guesswork into systematic problem-solving with proven strategies, powerful tools, and methodical approaches.
+Debugging is the search for the smallest true statement that explains a
+symptom. Every step below either makes the search space smaller or makes an
+observation more trustworthy. If a step does neither, skip it.
 
-## Use this skill when
+Platform tools live elsewhere: `debugging-instruments` covers LLDB, the memory
+graph and Instruments on Apple platforms; `agent-introspection-debugging` covers
+an AI agent debugging its own failed runs. This skill is the method they plug
+into.
 
-- Tracking down elusive bugs
-- Investigating performance issues
-- Debugging production incidents
-- Analyzing crash dumps or stack traces
-- Debugging distributed systems
+## The loop
 
-## Do not use this skill when
+```
+observe -> reproduce -> shrink -> hypothesise -> test one thing -> conclude
+    ^                                                   |
+    +-------------------- wrong? ----------------------+
+```
 
-- There is no reproducible issue or observable symptom
-- The task is purely feature development
-- You cannot access logs, traces, or runtime signals
+Write each hypothesis and each result down as you go, in a scratch file or the
+ticket. Most wasted debugging time is re-testing something already ruled out.
 
-## Instructions
+## 1. Pin down the symptom
 
-- Reproduce the issue and capture logs, traces, and environment details.
-- Form hypotheses and design controlled experiments.
-- Narrow scope with binary search and targeted instrumentation.
-- Document findings and verify the fix.
+Get the exact observable, not a paraphrase:
 
+- The full error text and stack trace, the input that produced it, the version
+  or commit, the environment (OS, runtime, config flags, locale, time zone).
+- What was expected instead, stated so it can be checked.
+- Whether it ever worked. If yes, when did it last work? That turns a search
+  into a bisection.
+
+"The order total is wrong" is a complaint. "An order with two discount codes
+totals 18.00 instead of 16.20 on build 4.12.0, first seen after Tuesday's
+release" is a symptom.
+
+## 2. Reproduce on demand
+
+A bug you cannot trigger is a bug you cannot confirm fixed. Push toward a
+reproduction that runs in seconds:
+
+1. Replay the exact input (request body, file, event payload) locally.
+2. If it does not reproduce, list the differences between your run and the
+   failing one: data, config, versions, concurrency, clock, network. Remove them
+   one at a time until it does.
+3. Turn the reproduction into an automated test as soon as it exists. It is the
+   regression test later.
+
+For an intermittent failure, loop it: run the test a few hundred times and count
+failures. A rate is an observation you can compare before and after a change.
+
+## 3. Shrink the failing case
+
+A smaller case has fewer places for the bug to hide.
+
+- **Input shrinking.** Halve the input and keep the half that still fails.
+  Repeat. Two hundred order lines become the three that matter. Property-based
+  testing tools do this automatically when they find a failure.
+- **Code shrinking.** Stub out collaborators one by one. If the failure survives
+  with the database replaced by an in-memory fake, the database is cleared.
+- **Environment shrinking.** Disable caches, feature flags, background jobs.
+
+Stop when removing anything else makes the failure disappear. What is left is
+the cause or points straight at it.
+
+## 4. Hypothesise, then try to kill each hypothesis
+
+Write two or three candidate explanations, each specific enough to predict
+something: "if the rounding happens per line instead of per order, then an order
+with a single line will total correctly." Then run the experiment that could
+prove it wrong.
+
+Rules that keep this honest:
+
+- Change one variable per experiment.
+- Predict the outcome before running it. A result you did not predict means your
+  model of the system is wrong, which is itself useful.
+- Prefer the cheap experiment that splits the hypotheses over the expensive one
+  that confirms your favourite.
+
+## Details on demand
+
+Load `references/hard-cases.md` for step 5 (bisecting commits, inputs and config), step 8 (production and cross-service bugs) and step 9 (timing, concurrency and memory bugs).
+
+## 6. Instrument with intent
+
+Add logging or breakpoints to answer a specific question, not to "see what
+happens".
+
+- Log the values that distinguish your hypotheses, at the boundary where they
+  enter and leave the suspect code.
+- Include an identifier (order id, request id) in every line so a single case
+  can be followed.
+- Conditional breakpoints and watchpoints beat stepping through a loop by hand.
+- Remove temporary instrumentation before committing, or promote it to proper
+  structured logging if it would have helped in production.
+
+## 7. Differential debugging
+
+When one run works and another fails, the cause is somewhere in the difference.
+Capture both and diff them:
+
+- Logs or traces of the same request on the good and bad build.
+- Environment dumps (`env`, dependency lock files, config) from the working and
+  failing machines.
+- Serialised state before the failing step on both paths.
+
+Normalise noise (timestamps, ids, addresses) first, then diff. The first
+unexpected divergence is where to look.
+
+## 10. Conclude properly
+
+A bug is fixed when:
+
+1. You can explain the mechanism in a sentence, including why it did not fail
+   before.
+2. The reproduction test fails without the fix and passes with it.
+3. The test sits in the suite at the level closest to the defect (a unit test
+   for a rounding rule, an integration test for a wrong query).
+4. You checked for siblings: the same pattern elsewhere in the codebase.
+
+If you cannot explain the mechanism, you have a change that makes the symptom go
+away, not a fix. Say so in the pull request.
+
+## When stuck
+
+- Explain the problem aloud or in writing to someone, step by step. Gaps in the
+  explanation are where the assumption is wrong.
+- Re-read the actual error message and the actual code, not your memory of them.
+- Question the "impossible": the file you edited is not the one being run, the
+  cache is stale, the build did not pick up the change, the config you read is
+  overridden later.
+- Take a break with the experiment log written down. It survives the break;
+  your working memory does not.

package/pipeline/skills/shared/external/debugging-strategies/references/hard-cases.md

@@ -0,0 +1,54 @@
+# Bisecting, production and timing bugs
+
+Companion to `../SKILL.md` for the `debugging-strategies` skill.
+
+## 5. Bisect
+
+When something used to work, bisection finds the change in log2(n) steps.
+
+**Across commits:**
+
+```bash
+git bisect start
+git bisect bad HEAD
+git bisect good v4.11.0
+git bisect run ./scripts/repro-total.sh   # exit 0 = good, 1 = bad, 125 = skip
+git bisect reset
+```
+
+The script is your reproduction from step 2. Use exit code 125 for commits that
+do not build, so bisect skips them.
+
+**Across other axes:** dependency versions (pin half of the upgraded packages
+back), configuration (revert half the changed keys), data (split the dataset),
+time (replay half the event log). Anything ordered or divisible can be bisected.
+
+## 8. Bugs that only show up in production or across services
+
+- **Correlation ids.** Every inbound request gets an id that is propagated to
+  downstream calls, queue messages and log lines. Without it you are joining
+  logs by timestamp, which lies under load.
+- **Distributed traces.** A trace shows which hop added the latency or returned
+  the error. Look for the first span that deviates, not the one that finally
+  threw.
+- **Recent change inventory.** Deploys, config pushes, feature-flag flips,
+  dependency upgrades, certificate rotations and traffic changes in the window
+  before the first failure. Most incidents line up with one of them.
+- **Reproduce with production data safely.** Capture the offending payload,
+  scrub personal data, replay it in a lower environment.
+- **Mitigate first.** In a live incident, roll back or flag off before root
+  causing. The investigation continues on the captured evidence.
+
+## 9. Heisenbugs: timing, concurrency, memory
+
+Bugs that move when observed usually involve ordering.
+
+- Suspect shared mutable state, missing awaits, callbacks on unexpected threads,
+  caches read while being written, and assumptions about event order.
+- Make timing deterministic: inject the clock, fix random seeds, run on a single
+  thread or a controlled executor, add artificial delays at the suspected race
+  window to widen it.
+- Use the platform's race and memory detectors (thread sanitizer, address
+  sanitizer, the runtime's race detector) on the looped reproduction.
+- Logging changes timing. Prefer in-memory ring buffers dumped on failure over
+  synchronous log writes inside the hot path.

package/pipeline/skills/shared/external/device-integrity/SKILL.md

@@ -1,6 +1,8 @@
 ---
 name: device-integrity
 description: "Verify device legitimacy and app integrity using DeviceCheck (DCDevice per-device bits) and App Attest (DCAppAttestService key generation, attestation, and assertion flows). Use when implementing fraud prevention, detecting compromised devices, validating app authenticity with Apple's servers, protecting sensitive API endpoints with attested requests, or adding device verification to a backend architecture."
+metadata:
+  source: "dpearson2699/swift-ios-skills (PolyForm Perimeter 1.0.0)"
 ---
 
 # Device Integrity