softwareobservatory 0.2.1 → 0.3.1

package/data/sensors.json

@@ -1,4 +1,5 @@
 {
+  "attribution": "Software Observatory by Justin Abrahms — https://softwareobservatory.com. Licensed CC BY-SA 4.0; adaptations must carry the same license.",
   "families": [
     {
       "count": 9,
@@ -82,6 +83,8 @@
     }
   ],
   "generated_at": "2026-08-24T00:00:00Z",
+  "license": "CC-BY-SA-4.0",
+  "license_url": "https://creativecommons.org/licenses/by-sa/4.0/",
   "sensors": [
     {
       "body_html": "<p>Is the artifact you are about to deploy structurally the one your pipeline\nbuilt? A software bill of materials plus build provenance attestation\n(SLSA-style) answers: these sources, these dependencies, this builder, this\nhash. A structural sensor aimed at the moment of deployment, where the\nartifact leaves the world you control.</p>\n<h2>In practice</h2>\n<p>A reading is two artifacts: the SBOM, and the attestation tying it to\nthe artifact you are about to deploy.</p>\n<pre><code>artifact:   payments-api@sha256:9f1c2b...\nbuilder:    https://ci.internal/builders/release (SLSA level 3)\nbuild:      run 8841, commit a41d2e9, 2026-08-19T14:02Z\nsignature:  verified, sigstore keyless\n\ncomponents:\n  log4j-core        2.14.1  pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1\n  jackson-databind  2.15.2  pkg:maven/com.fasterxml.jackson.core/jackson-databind@2.15.2\n</code></pre>\n<p>Cross-referenced against the vulnerability feed, the same reading says:</p>\n<table>\n<thead>\n<tr>\n<th>Component</th>\n<th>Version</th>\n<th>Finding</th>\n<th>Severity</th>\n<th>Verdict</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>log4j-core</td>\n<td>2.14.1</td>\n<td>CVE-2021-44228, remote code execution</td>\n<td>critical</td>\n<td>block deploy</td>\n</tr>\n<tr>\n<td>jackson-databind</td>\n<td>2.15.2</td>\n<td>no known CVEs</td>\n<td>none</td>\n<td>clear</td>\n</tr>\n</tbody>\n</table>\n<p>Reading it well:</p>\n<ul>\n<li><strong>Verify the signature before reading the contents.</strong> An SBOM you\n  cannot tie to the builder is a claim, not an attestation. The hash\n  comparison is the whole sensor; everything else is commentary.</li>\n<li><strong>Read versions, not names.</strong> Two minor versions can be the\n  difference between clear and critical, and name-only matching misses\n  it.</li>\n<li><strong>Treat &ldquo;not in the SBOM&rdquo; as a finding.</strong> A component present in the\n  artifact but absent from the manifest means the SBOM was generated\n  from the wrong place, usually the source tree instead of the build.</li>\n</ul>\n<h2>Response playbook</h2>\n<p>When the sensor fires:</p>\n<ol>\n<li><strong>Block the deploy on any mismatch.</strong> Unverified signature, hash\n   mismatch, or missing attestation all mean the artifact does not\n   ship. Do not deploy first and reconcile later; the artifact leaves\n   the world you control at deploy time.</li>\n<li><strong>Block on critical vulnerability findings.</strong> A critical CVE in a\n   shipped component is a stop-ship condition. Confirm whether the\n   component is reachable in your build; if it is dead weight,\n   upgrade it anyway, because the next reader of the SBOM will not\n   know.</li>\n<li><strong>Rebuild from the attested source, not from the cached artifact.</strong>\n   A mismatch usually means the build ran somewhere else, or the\n   cache is stale. Re-run the pipeline and compare hashes again.</li>\n<li><strong>Escalate signature failures, not just dependency failures.</strong> A\n   vulnerable dependency is a fix; a bad or missing signature is a\n   possible supply-chain event. Notify security before retrying.</li>\n<li><strong>Record the attestation hash at deploy time.</strong> When a CVE ships\n   next quarter, you need to answer which deployments carry it, from\n   the SBOM, not from memory.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Malice inside the sources themselves, or a compromised builder that signs\nits own output faithfully. Provenance tells you the artifact is the one the\npipeline produced; whether the pipeline was honest is a question for\n<a class=\"body-link\" href=\"/sensors/static-security-analysis/\">static security analysis</a> and dependency\nauditing.</p>\n<p>Supply chain security is a deeper topic than this catalog covers.\nBuild provenance and SBOMs are one sensor in the structural family — they\nverify the artifact matches what the pipeline built — but the broader\npractice of supply chain security (dependency vulnerabilities, artifact\nsigning trust roots, runtime attestation, policy enforcement) deserves its\nown resources. See <a class=\"body-link\" href=\"https://slsa.dev\">SLSA</a> and\n<a class=\"body-link\" href=\"https://sigstore.dev\">Sigstore</a> for dedicated treatment.</p>",
@@ -154,7 +157,8 @@
       "see_also_ids": [
         "SO-001",
         "SO-014",
-        "SO-012b"
+        "SO-012b",
+        "SO-014d"
       ],
       "see_also_pages": [],
       "slug": "build-provenance-sbom",
@@ -664,8 +668,8 @@
       "url_path": "sensors/compiler/"
     },
     {
-      "body_html": "<p>Behavioral guarantees checked before the code ever runs. Typestate systems,\nrefinement types, and design-by-contract annotations (Eiffel-style\npre/postconditions, Dafny <code>requires</code>/<code>ensures</code>) ask the compiler to prove\nthat certain behaviors are impossible, not merely unlikely.</p>\n<p>Where a <a class=\"body-link\" href=\"/sensors/type-checker/\">type checker</a> answers &ldquo;do the shapes fit?&rdquo;, a\nrefinement type answers &ldquo;can <code>withdraw(amount)</code> ever be called with\n<code>amount &gt; balance</code>?&rdquo; — a behavioral claim, discharged at build time.</p>\n<h2>In practice</h2>\n<p>A contract check reads like a proof obligation returned unpaid. The\ndiagnostic names the obligation that failed, and usually the related\nlocation where it was stated:</p>\n<pre><code>Program.dfy(14,4): Error: A precondition for this call could not be\nproved on an entry point of this program\nProgram.dfy(14,17): Related location: This is the precondition that\ncould not be proved\nProgram.dfy(9,11): Related location: this is the precondition\n</code></pre>\n<p>Three causes produce the same message, and the fix is different for\neach: the implementation violates its own contract, the contract is\nwrong, or the prover needs an intermediate lemma to connect the two.\nThe message alone does not say which. Unlike a test failure, though,\nthe verdict is deterministic: a contract that fails today fails on\nevery machine, and a proof that discharges never flakes. The counter-\nexamples some checkers print are worth keeping; they are the minimal\ninputs that break the claim.</p>\n<h2>Response playbook</h2>\n<p>When a contract fails to discharge:</p>\n<ol>\n<li><strong>Reproduce with the smallest failing input.</strong> If the checker\n   emits a counter-example, run it; if not, derive one from the\n   failed obligation.</li>\n<li><strong>Decide whether the implementation or the contract is wrong.</strong>\n   A failed proof is a genuine disagreement between two claims the\n   author made, and one of them must be retracted.</li>\n<li><strong>If both are right, supply the missing lemma.</strong> Splitting the\n   obligation into smaller steps is how proofs go through; deleting\n   the contract is how they get abandoned.</li>\n<li><strong>Demote what cannot be proved.</strong> An obligation that resists the\n   prover can still ship as a\n   <a class=\"body-link\" href=\"/sensors/runtime-invariants/\">runtime invariant</a>, checked on every\n   execution instead of all executions.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<p>The checker cannot be gamed, but the contracts are written by the\nsame mind that writes the code, so the specification itself can be\ndegraded:</p>\n<ul>\n<li><strong>Trivial contracts.</strong> <code>ensures true</code>, postconditions that restate\n  the type signature. The checker passes and protects nothing.</li>\n<li><strong>Weaken until green.</strong> Each fight loosens the precondition one\n  notch until the proof goes through by erosion rather than by\n  correctness.</li>\n<li><strong>Verification off the merge path.</strong> A proof job that runs nightly\n  or on demand, rather than on every merge, is a proof nobody is\n  waiting for.</li>\n</ul>\n<p>The meta-signal is contract strength: sample annotated functions and\ncount how many have postconditions that are trivially true.</p>\n<h2>What it cannot detect</h2>\n<p>Contracts only cover what was specified. The specification itself is\nwritten by the same mind that wrote the code, which is why contract checking\ncomposes with rather than replaces <a class=\"body-link\" href=\"/sensors/example-based-tests/\">example-based tests</a>\nand <a class=\"body-link\" href=\"/sensors/fuzzing/\">property testing</a>.</p>",
-      "body_text": "Behavioral guarantees checked before the code ever runs. Typestate systems, refinement types, and design-by-contract annotations (Eiffel-style pre/postconditions, Dafny requires / ensures ) ask the compiler to prove that certain behaviors are impossible, not merely unlikely. Where a type checker answers “do the shapes fit?”, a refinement type answers “can withdraw(amount) ever be called with amount > balance ?” — a behavioral claim, discharged at build time. In practice A contract check reads like a proof obligation returned unpaid. The diagnostic names the obligation that failed, and usually the related location where it was stated: Program.dfy(14,4): Error: A precondition for this call could not be proved on an entry point of this program Program.dfy(14,17): Related location: This is the precondition that could not be proved Program.dfy(9,11): Related location: this is the precondition Three causes produce the same message, and the fix is different for each: the implementation violates its own contract, the contract is wrong, or the prover needs an intermediate lemma to connect the two. The message alone does not say which. Unlike a test failure, though, the verdict is deterministic: a contract that fails today fails on every machine, and a proof that discharges never flakes. The counter- examples some checkers print are worth keeping; they are the minimal inputs that break the claim. Response playbook When a contract fails to discharge: Reproduce with the smallest failing input. If the checker emits a counter-example, run it; if not, derive one from the failed obligation. Decide whether the implementation or the contract is wrong. A failed proof is a genuine disagreement between two claims the author made, and one of them must be retracted. If both are right, supply the missing lemma. Splitting the obligation into smaller steps is how proofs go through; deleting the contract is how they get abandoned. Demote what cannot be proved. An obligation that resists the prover can still ship as a runtime invariant , checked on every execution instead of all executions. How it gets gamed The checker cannot be gamed, but the contracts are written by the same mind that writes the code, so the specification itself can be degraded: Trivial contracts. ensures true , postconditions that restate the type signature. The checker passes and protects nothing. Weaken until green. Each fight loosens the precondition one notch until the proof goes through by erosion rather than by correctness. Verification off the merge path. A proof job that runs nightly or on demand, rather than on every merge, is a proof nobody is waiting for. The meta-signal is contract strength: sample annotated functions and count how many have postconditions that are trivially true. What it cannot detect Contracts only cover what was specified. The specification itself is written by the same mind that wrote the code, which is why contract checking composes with rather than replaces example-based tests and property testing .",
+      "body_html": "<p>Behavioral guarantees checked before the code ever runs. Typestate systems,\nrefinement types, and design-by-contract annotations (Eiffel-style\npre/postconditions, Dafny <code>requires</code>/<code>ensures</code>) ask the compiler to prove\nthat certain behaviors are impossible, not merely unlikely.</p>\n<p>Where a <a class=\"body-link\" href=\"/sensors/type-checker/\">type checker</a> answers &ldquo;do the shapes fit?&rdquo;, a\nrefinement type answers &ldquo;can <code>withdraw(amount)</code> ever be called with\n<code>amount &gt; balance</code>?&rdquo; — a behavioral claim, discharged at build time.</p>\n<h2>In practice</h2>\n<p>A contract check reads like a proof obligation returned unpaid. The\ndiagnostic names the obligation that failed, and usually the related\nlocation where it was stated:</p>\n<pre><code>Program.dfy(14,4): Error: A precondition for this call could not be\nproved on an entry point of this program\nProgram.dfy(14,17): Related location: This is the precondition that\ncould not be proved\nProgram.dfy(9,11): Related location: this is the precondition\n</code></pre>\n<p>Three causes produce the same message, and the fix is different for\neach: the implementation violates its own contract, the contract is\nwrong, or the prover needs an intermediate lemma to connect the two.\nThe message alone does not say which. Unlike a test failure, though,\nthe verdict is deterministic: a contract that fails today fails on\nevery machine, and a proof that discharges never flakes. What the\nmessage will not hand you is a failing input. Dafny can extract one\nfrom the solver with <code>--extract-counterexample</code>, and says in the same\nbreath that it &ldquo;cannot guarantee that the counterexample it reports\nprovably violates the assertion&rdquo;, and that the output &ldquo;should be\ninspected manually and treated as a hint&rdquo;. A solver model is not a\nminimal input and is not always a reachable one.</p>\n<h2>Response playbook</h2>\n<p>When a contract fails to discharge:</p>\n<ol>\n<li><strong>Try to reproduce with a concrete input.</strong> If the checker will\n   extract a counter-example, run it before believing it — the model\n   is a hint, not a witness. If it will not, derive a candidate from\n   the failed obligation by hand.</li>\n<li><strong>Decide whether the implementation or the contract is wrong.</strong>\n   A failed proof is a genuine disagreement between two claims the\n   author made, and one of them must be retracted.</li>\n<li><strong>If both are right, supply the missing lemma.</strong> Splitting the\n   obligation into smaller steps is how proofs go through; deleting\n   the contract is how they get abandoned.</li>\n<li><strong>Demote what cannot be proved.</strong> An obligation that resists the\n   prover can still ship as a\n   <a class=\"body-link\" href=\"/sensors/runtime-invariants/\">runtime invariant</a>, checked on every\n   execution instead of all executions.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<p>The checker cannot be gamed, but the contracts are written by the\nsame mind that writes the code, so the specification itself can be\ndegraded:</p>\n<ul>\n<li><strong>Trivial contracts.</strong> <code>ensures true</code>, postconditions that restate\n  the type signature. The checker passes and protects nothing.</li>\n<li><strong>Weaken until green.</strong> Each fight loosens the precondition one\n  notch until the proof goes through by erosion rather than by\n  correctness.</li>\n<li><strong>Verification off the merge path.</strong> A proof job that runs nightly\n  or on demand, rather than on every merge, is a proof nobody is\n  waiting for.</li>\n</ul>\n<p>The meta-signal is contract strength: sample annotated functions and\ncount how many have postconditions that are trivially true.</p>\n<h2>What it cannot detect</h2>\n<p>Contracts only cover what was specified. The specification itself is\nwritten by the same mind that wrote the code, which is why contract checking\ncomposes with rather than replaces <a class=\"body-link\" href=\"/sensors/example-based-tests/\">example-based tests</a>\nand <a class=\"body-link\" href=\"/sensors/fuzzing/\">property testing</a>.</p>",
+      "body_text": "Behavioral guarantees checked before the code ever runs. Typestate systems, refinement types, and design-by-contract annotations (Eiffel-style pre/postconditions, Dafny requires / ensures ) ask the compiler to prove that certain behaviors are impossible, not merely unlikely. Where a type checker answers “do the shapes fit?”, a refinement type answers “can withdraw(amount) ever be called with amount > balance ?” — a behavioral claim, discharged at build time. In practice A contract check reads like a proof obligation returned unpaid. The diagnostic names the obligation that failed, and usually the related location where it was stated: Program.dfy(14,4): Error: A precondition for this call could not be proved on an entry point of this program Program.dfy(14,17): Related location: This is the precondition that could not be proved Program.dfy(9,11): Related location: this is the precondition Three causes produce the same message, and the fix is different for each: the implementation violates its own contract, the contract is wrong, or the prover needs an intermediate lemma to connect the two. The message alone does not say which. Unlike a test failure, though, the verdict is deterministic: a contract that fails today fails on every machine, and a proof that discharges never flakes. What the message will not hand you is a failing input. Dafny can extract one from the solver with --extract-counterexample , and says in the same breath that it “cannot guarantee that the counterexample it reports provably violates the assertion”, and that the output “should be inspected manually and treated as a hint”. A solver model is not a minimal input and is not always a reachable one. Response playbook When a contract fails to discharge: Try to reproduce with a concrete input. If the checker will extract a counter-example, run it before believing it — the model is a hint, not a witness. If it will not, derive a candidate from the failed obligation by hand. Decide whether the implementation or the contract is wrong. A failed proof is a genuine disagreement between two claims the author made, and one of them must be retracted. If both are right, supply the missing lemma. Splitting the obligation into smaller steps is how proofs go through; deleting the contract is how they get abandoned. Demote what cannot be proved. An obligation that resists the prover can still ship as a runtime invariant , checked on every execution instead of all executions. How it gets gamed The checker cannot be gamed, but the contracts are written by the same mind that writes the code, so the specification itself can be degraded: Trivial contracts. ensures true , postconditions that restate the type signature. The checker passes and protects nothing. Weaken until green. Each fight loosens the precondition one notch until the proof goes through by erosion rather than by correctness. Verification off the merge path. A proof job that runs nightly or on demand, rather than on every merge, is a proof nobody is waiting for. The meta-signal is contract strength: sample annotated functions and count how many have postconditions that are trivially true. What it cannot detect Contracts only cover what was specified. The specification itself is written by the same mind that wrote the code, which is why contract checking composes with rather than replaces example-based tests and property testing .",
       "family": "structural",
       "frontmatter": {
         "actionability": "blocking",
@@ -703,6 +707,11 @@
             "title": "Dafny",
             "url": "https://dafny.org"
           },
+          {
+            "kind": "other",
+            "title": "Dafny user guide, verification debugging",
+            "url": "https://dafny.org/latest/DafnyRef/DafnyRef#sec-counterexamples"
+          },
           {
             "description": "Static analysis and verification for C",
             "kind": "tool",
@@ -734,7 +743,8 @@
         "SO-001",
         "SO-001d",
         "SO-002",
-        "SO-005"
+        "SO-005",
+        "SO-017"
       ],
       "see_also_pages": [],
       "slug": "contract-refinement-types",
@@ -744,6 +754,7 @@
         "abandoned",
         "all",
         "alone",
+        "always",
         "amount",
         "and",
         "annotated",
@@ -751,33 +762,36 @@
         "answers",
         "are",
         "ask",
+        "assertion",
         "author",
         "balance",
         "based",
         "before",
         "behavioral",
         "behaviors",
+        "believing",
         "between",
         "both",
-        "break",
+        "breath",
         "build",
         "but",
         "call",
         "called",
         "can",
+        "candidate",
         "cannot",
         "causes",
         "certain",
         "check",
         "checked",
         "checker",
-        "checkers",
         "checking",
         "claim",
         "claims",
         "code",
         "compiler",
         "composes",
+        "concrete",
         "connect",
         "contract",
         "contracts",
@@ -785,6 +799,7 @@
         "could",
         "count",
         "counter",
+        "counterexample",
         "cover",
         "dafny",
         "decide",
@@ -806,7 +821,6 @@
         "does",
         "each",
         "eiffel",
-        "emits",
         "ensures",
         "entry",
         "erosion",
@@ -814,9 +828,9 @@
         "ever",
         "every",
         "example",
-        "examples",
         "execution",
         "executions",
+        "extract",
         "failed",
         "failing",
         "fails",
@@ -834,13 +848,16 @@
         "gets",
         "goes",
         "green",
+        "guarantee",
         "guarantees",
+        "hand",
         "have",
+        "hint",
         "how",
         "implementation",
         "impossible",
         "input",
-        "inputs",
+        "inspected",
         "instead",
         "intermediate",
         "into",
@@ -848,13 +865,13 @@
         "its",
         "itself",
         "job",
-        "keeping",
         "lemma",
         "like",
         "location",
         "loosens",
         "machine",
         "made",
+        "manually",
         "many",
         "merely",
         "merge",
@@ -863,6 +880,7 @@
         "mind",
         "minimal",
         "missing",
+        "model",
         "must",
         "names",
         "needs",
@@ -876,6 +894,7 @@
         "off",
         "one",
         "only",
+        "output",
         "own",
         "passes",
         "path",
@@ -885,21 +904,23 @@
         "practice",
         "pre",
         "precondition",
-        "print",
         "produce",
         "program",
         "proof",
         "proofs",
         "property",
         "protects",
+        "provably",
         "prove",
         "proved",
         "prover",
         "rather",
+        "reachable",
         "reads",
         "refinement",
         "related",
         "replaces",
+        "reports",
         "reproduce",
         "requires",
         "resists",
@@ -914,13 +935,14 @@
         "same",
         "sample",
         "say",
+        "says",
         "shapes",
         "ship",
+        "should",
         "signal",
         "signature",
         "smaller",
-        "smallest",
-        "some",
+        "solver",
         "specification",
         "specified",
         "splitting",
@@ -945,9 +967,11 @@
         "through",
         "time",
         "today",
+        "treated",
         "trivial",
         "trivially",
         "true",
+        "try",
         "two",
         "type",
         "types",
@@ -969,13 +993,15 @@
         "whether",
         "which",
         "why",
+        "will",
         "with",
         "withdraw",
-        "worth",
+        "witness",
         "writes",
         "written",
         "wrong",
-        "wrote"
+        "wrote",
+        "you"
       ],
       "url_path": "sensors/contract-refinement-types/"
     },
@@ -1292,7 +1318,8 @@
         "SO-012",
         "SO-013",
         "SO-001",
-        "SO-001e"
+        "SO-001e",
+        "SO-017"
       ],
       "see_also_pages": [],
       "slug": "model-checking",
@@ -3199,8 +3226,8 @@
       "url_path": "sensors/contract-tests/"
     },
     {
-      "body_html": "<p>Given X, expect Y. The fundamental behavioral sensor.</p>\n<p>Fundamentally different from <a class=\"body-link\" href=\"/sensors/diff-coverage/\">coverage</a>: coverage says\n&ldquo;this code executed.&rdquo; A behavioral assertion says &ldquo;this code produced the\n<em>right result</em>.&rdquo; That is a massive distinction.</p>\n<h2>The weakness</h2>\n<p>Example-based tests are only as good as the examples chosen. They test what\nthe author thought to test. <a class=\"body-link\" href=\"/sensors/mutation-testing/\">Mutation testing</a> and\n<a class=\"body-link\" href=\"/sensors/metamorphic-testing/\">metamorphic testing</a> exist to find the gaps.</p>\n<h2>In practice</h2>\n<p>A reading is an assertion failure with both sides of the comparison\nspelled out:</p>\n<pre><code>FAILED tests/test_pricing.py::test_discount_applies_above_threshold\n\ntests/test_pricing.py:42: in test_discount_applies_above_threshold\n    assert price_with_discount(100, tier=&quot;gold&quot;) == 90.0\nE   AssertionError: assert 95.0 == 90.0\nE    +  where 95.0 = price_with_discount(100, tier='gold')\n\n================== 1 failed, 213 passed in 4.21s ==================\n</code></pre>\n<p>Reading it well takes three habits:</p>\n<ol>\n<li><strong>Read the assertion before the code.</strong> The failure shows two\n   values; the first question is which one is wrong. The assertion\n   encodes the intended behavior, and sometimes it is the test that\n   needs fixing, not the code.</li>\n<li><strong>One failure is a signal, not a ratio.</strong> &ldquo;1 failed, 213 passed&rdquo;\n   is not 99.5% healthy. The pass count is context; the failure is\n   the reading.</li>\n<li><strong>Separate the verdict from the harness.</strong> An assertion error is a\n   behavioral reading. A collection error, import error, or fixture\n   timeout is the harness breaking, which means the suite produced no\n   reading at all.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<p>The suite belongs to the same people as the code, so it can be bent:</p>\n<ul>\n<li><strong>Delete or skip the failure.</strong> <code>skip</code>, <code>xfail</code>, and &ldquo;I will fix\n  it tomorrow&rdquo; turn red to green without touching the code. A rising\n  skip count is the suite telling you it is being silenced.</li>\n<li><strong>Weaken the assertion.</strong> Replacing <code>== 90.0</code> with <code>is not None</code>\n  makes the test pass and the sensor blind. The test still runs, so\n  the suite looks healthy while detecting less.</li>\n<li><strong>Pin the symptom.</strong> Hard-coding the current output into the\n  expectation turns the test into a tautology that passes for any\n  implementation.</li>\n<li><strong>Sample only the happy path.</strong> Choosing examples that avoid the\n  buggy branch keeps the suite green and the bug alive. <a class=\"body-link\" href=\"/sensors/mutation-testing/\">Mutation\n  testing</a> is the sensor for this gap.</li>\n</ul>\n<p>The meta-signal is the skip count plus the share of test-file diffs\nthat weaken an assertion. Neither shows up in coverage.</p>\n<h2>Response playbook</h2>\n<p>When a test fails:</p>\n<ol>\n<li><strong>Reproduce it locally.</strong> A failure you cannot reproduce is not\n   yet a finding; it is a question. Get the exact input and the\n   exact assertion before anything else.</li>\n<li><strong>Decide which side is wrong.</strong> The assertion encodes intended\n   behavior. If the code violated it, fix the code. If the\n   expectation was wrong, fix the test and say why, because a\n   silently rewritten expectation is a deleted sensor.</li>\n<li><strong>Fix the cause, then re-run the full suite.</strong> A fix that breaks\n   a sibling test has revealed a second assumption you did not know\n   you had.</li>\n<li><strong>Add the neighbor cases.</strong> If <code>100</code> failed, the values just\n   above and below the boundary are now suspects. The failure is a\n   free tour of the edge of the behavior.</li>\n<li><strong>Never delete the test to unblock a merge.</strong> If the test is\n   wrong, rewrite it with the reason in the commit. If it is right,\n   the merge is not done.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Missing behavior, untested edge cases, and <a class=\"body-link\" href=\"/sensors/contract-tests/\">integration\nfailures</a> that emerge only when components are\nconnected.</p>",
-      "body_text": "Given X, expect Y. The fundamental behavioral sensor. Fundamentally different from coverage : coverage says “this code executed.” A behavioral assertion says “this code produced the right result .” That is a massive distinction. The weakness Example-based tests are only as good as the examples chosen. They test what the author thought to test. Mutation testing and metamorphic testing exist to find the gaps. In practice A reading is an assertion failure with both sides of the comparison spelled out: FAILED tests/test_pricing.py::test_discount_applies_above_threshold tests/test_pricing.py:42: in test_discount_applies_above_threshold assert price_with_discount(100, tier=\"gold\") == 90.0 E AssertionError: assert 95.0 == 90.0 E + where 95.0 = price_with_discount(100, tier='gold') ================== 1 failed, 213 passed in 4.21s ================== Reading it well takes three habits: Read the assertion before the code. The failure shows two values; the first question is which one is wrong. The assertion encodes the intended behavior, and sometimes it is the test that needs fixing, not the code. One failure is a signal, not a ratio. “1 failed, 213 passed” is not 99.5% healthy. The pass count is context; the failure is the reading. Separate the verdict from the harness. An assertion error is a behavioral reading. A collection error, import error, or fixture timeout is the harness breaking, which means the suite produced no reading at all. How it gets gamed The suite belongs to the same people as the code, so it can be bent: Delete or skip the failure. skip , xfail , and “I will fix it tomorrow” turn red to green without touching the code. A rising skip count is the suite telling you it is being silenced. Weaken the assertion. Replacing == 90.0 with is not None makes the test pass and the sensor blind. The test still runs, so the suite looks healthy while detecting less. Pin the symptom. Hard-coding the current output into the expectation turns the test into a tautology that passes for any implementation. Sample only the happy path. Choosing examples that avoid the buggy branch keeps the suite green and the bug alive. Mutation testing is the sensor for this gap. The meta-signal is the skip count plus the share of test-file diffs that weaken an assertion. Neither shows up in coverage. Response playbook When a test fails: Reproduce it locally. A failure you cannot reproduce is not yet a finding; it is a question. Get the exact input and the exact assertion before anything else. Decide which side is wrong. The assertion encodes intended behavior. If the code violated it, fix the code. If the expectation was wrong, fix the test and say why, because a silently rewritten expectation is a deleted sensor. Fix the cause, then re-run the full suite. A fix that breaks a sibling test has revealed a second assumption you did not know you had. Add the neighbor cases. If 100 failed, the values just above and below the boundary are now suspects. The failure is a free tour of the edge of the behavior. Never delete the test to unblock a merge. If the test is wrong, rewrite it with the reason in the commit. If it is right, the merge is not done. What it cannot detect Missing behavior, untested edge cases, and integration failures that emerge only when components are connected.",
+      "body_html": "<p>Given X, expect Y. The fundamental behavioral sensor.</p>\n<p>Fundamentally different from <a class=\"body-link\" href=\"/sensors/diff-coverage/\">coverage</a>: coverage says\n&ldquo;this code executed.&rdquo; A behavioral assertion says &ldquo;this code produced the\n<em>right result</em>.&rdquo; That is a massive distinction.</p>\n<h2>The weakness</h2>\n<p>Example-based tests are only as good as the examples chosen. They test what\nthe author thought to test. <a class=\"body-link\" href=\"/sensors/property-based-testing/\">Property-based\ntesting</a>, <a class=\"body-link\" href=\"/sensors/mutation-testing/\">mutation\ntesting</a> and <a class=\"body-link\" href=\"/sensors/metamorphic-testing/\">metamorphic\ntesting</a> exist to find the gaps.</p>\n<h2>In practice</h2>\n<p>A reading is an assertion failure with both sides of the comparison\nspelled out:</p>\n<pre><code>FAILED tests/test_pricing.py::test_discount_applies_above_threshold\n\ntests/test_pricing.py:42: in test_discount_applies_above_threshold\n    assert price_with_discount(100, tier=&quot;gold&quot;) == 90.0\nE   AssertionError: assert 95.0 == 90.0\nE    +  where 95.0 = price_with_discount(100, tier='gold')\n\n================== 1 failed, 213 passed in 4.21s ==================\n</code></pre>\n<p>Reading it well takes three habits:</p>\n<ol>\n<li><strong>Read the assertion before the code.</strong> The failure shows two\n   values; the first question is which one is wrong. The assertion\n   encodes the intended behavior, and sometimes it is the test that\n   needs fixing, not the code.</li>\n<li><strong>One failure is a signal, not a ratio.</strong> &ldquo;1 failed, 213 passed&rdquo;\n   is not 99.5% healthy. The pass count is context; the failure is\n   the reading.</li>\n<li><strong>Separate the verdict from the harness.</strong> An assertion error is a\n   behavioral reading. A collection error, import error, or fixture\n   timeout is the harness breaking, which means the suite produced no\n   reading at all.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<p>The suite belongs to the same people as the code, so it can be bent:</p>\n<ul>\n<li><strong>Delete or skip the failure.</strong> <code>skip</code>, <code>xfail</code>, and &ldquo;I will fix\n  it tomorrow&rdquo; turn red to green without touching the code. A rising\n  skip count is the suite telling you it is being silenced.</li>\n<li><strong>Weaken the assertion.</strong> Replacing <code>== 90.0</code> with <code>is not None</code>\n  makes the test pass and the sensor blind. The test still runs, so\n  the suite looks healthy while detecting less.</li>\n<li><strong>Pin the symptom.</strong> Hard-coding the current output into the\n  expectation turns the test into a tautology that passes for any\n  implementation.</li>\n<li><strong>Sample only the happy path.</strong> Choosing examples that avoid the\n  buggy branch keeps the suite green and the bug alive. <a class=\"body-link\" href=\"/sensors/mutation-testing/\">Mutation\n  testing</a> is the sensor for this gap.</li>\n</ul>\n<p>The meta-signal is the skip count plus the share of test-file diffs\nthat weaken an assertion. Neither shows up in coverage.</p>\n<h2>Response playbook</h2>\n<p>When a test fails:</p>\n<ol>\n<li><strong>Reproduce it locally.</strong> A failure you cannot reproduce is not\n   yet a finding; it is a question. Get the exact input and the\n   exact assertion before anything else.</li>\n<li><strong>Decide which side is wrong.</strong> The assertion encodes intended\n   behavior. If the code violated it, fix the code. If the\n   expectation was wrong, fix the test and say why, because a\n   silently rewritten expectation is a deleted sensor.</li>\n<li><strong>Fix the cause, then re-run the full suite.</strong> A fix that breaks\n   a sibling test has revealed a second assumption you did not know\n   you had.</li>\n<li><strong>Add the neighbor cases.</strong> If <code>100</code> failed, the values just\n   above and below the boundary are now suspects. The failure is a\n   free tour of the edge of the behavior.</li>\n<li><strong>Never delete the test to unblock a merge.</strong> If the test is\n   wrong, rewrite it with the reason in the commit. If it is right,\n   the merge is not done.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Missing behavior, untested edge cases, and <a class=\"body-link\" href=\"/sensors/contract-tests/\">integration\nfailures</a> that emerge only when components are\nconnected.</p>",
+      "body_text": "Given X, expect Y. The fundamental behavioral sensor. Fundamentally different from coverage : coverage says “this code executed.” A behavioral assertion says “this code produced the right result .” That is a massive distinction. The weakness Example-based tests are only as good as the examples chosen. They test what the author thought to test. Property-based testing , mutation testing and metamorphic testing exist to find the gaps. In practice A reading is an assertion failure with both sides of the comparison spelled out: FAILED tests/test_pricing.py::test_discount_applies_above_threshold tests/test_pricing.py:42: in test_discount_applies_above_threshold assert price_with_discount(100, tier=\"gold\") == 90.0 E AssertionError: assert 95.0 == 90.0 E + where 95.0 = price_with_discount(100, tier='gold') ================== 1 failed, 213 passed in 4.21s ================== Reading it well takes three habits: Read the assertion before the code. The failure shows two values; the first question is which one is wrong. The assertion encodes the intended behavior, and sometimes it is the test that needs fixing, not the code. One failure is a signal, not a ratio. “1 failed, 213 passed” is not 99.5% healthy. The pass count is context; the failure is the reading. Separate the verdict from the harness. An assertion error is a behavioral reading. A collection error, import error, or fixture timeout is the harness breaking, which means the suite produced no reading at all. How it gets gamed The suite belongs to the same people as the code, so it can be bent: Delete or skip the failure. skip , xfail , and “I will fix it tomorrow” turn red to green without touching the code. A rising skip count is the suite telling you it is being silenced. Weaken the assertion. Replacing == 90.0 with is not None makes the test pass and the sensor blind. The test still runs, so the suite looks healthy while detecting less. Pin the symptom. Hard-coding the current output into the expectation turns the test into a tautology that passes for any implementation. Sample only the happy path. Choosing examples that avoid the buggy branch keeps the suite green and the bug alive. Mutation testing is the sensor for this gap. The meta-signal is the skip count plus the share of test-file diffs that weaken an assertion. Neither shows up in coverage. Response playbook When a test fails: Reproduce it locally. A failure you cannot reproduce is not yet a finding; it is a question. Get the exact input and the exact assertion before anything else. Decide which side is wrong. The assertion encodes intended behavior. If the code violated it, fix the code. If the expectation was wrong, fix the test and say why, because a silently rewritten expectation is a deleted sensor. Fix the cause, then re-run the full suite. A fix that breaks a sibling test has revealed a second assumption you did not know you had. Add the neighbor cases. If 100 failed, the values just above and below the boundary are now suspects. The failure is a free tour of the edge of the behavior. Never delete the test to unblock a merge. If the test is wrong, rewrite it with the reason in the commit. If it is right, the merge is not done. What it cannot detect Missing behavior, untested edge cases, and integration failures that emerge only when components are connected.",
       "family": "behavioral",
       "frontmatter": {
         "actionability": "guiding",
@@ -3275,7 +3302,8 @@
       "see_also_ids": [
         "SO-002c",
         "SO-002d",
-        "SO-003"
+        "SO-003",
+        "SO-005e"
       ],
       "see_also_pages": [],
       "slug": "example-based-tests",
@@ -3431,6 +3459,7 @@
         "practice",
         "price_with_discount",
         "produced",
+        "property",
         "question",
         "ratio",
         "read",
@@ -4483,7 +4512,8 @@
       "see_also_ids": [
         "SO-006c",
         "SO-007",
-        "SO-012b"
+        "SO-012b",
+        "SO-006e"
       ],
       "see_also_pages": [],
       "slug": "synthetic-monitoring",
@@ -6045,6 +6075,8 @@
       ],
       "see_also_ids": [
         "SO-003b",
+        "SO-005e",
+        "SO-012d",
         "SO-005",
         "SO-005b"
       ],
@@ -7810,8 +7842,8 @@
       "url_path": "sensors/runtime-invariants/"
     },
     {
-      "body_html": "<p>Invariants the compiler refuses to let you violate: Dafny <code>invariant</code>\nclauses, Frama-C annotations, JML specs, type-level witnesses like\n<code>NonEmptyList</code>. Where <a class=\"body-link\" href=\"/sensors/database-invariants/\">database invariants</a> are\nchecked by a live system against live data, these are proved once, at build\ntime, for all possible executions.</p>\n<h2>In practice</h2>\n<p>The reading is the same as a <a class=\"body-link\" href=\"/sensors/type-checker/\">type-checker</a> error,\nraised against a property of the program instead of a type:</p>\n<pre><code>src/Queue.hs:23:1: Error: Liquid Type Mismatch\n  Invariant `size q &gt;= 0` could not be proved\n  Counter-example: `q` after `dequeue (mkQueue [])`\n</code></pre>\n<p>Or, at the type level: a compiler that refuses <code>head []</code> because the\nlist&rsquo;s type carries a proof of non-emptiness. Either way the verdict\narrives at build time, once, for all possible executions, and a\ncounter-example when the checker can find one.</p>\n<p>The counter-example is the part to read first: it is the smallest\nstate that violates the invariant, and it usually says immediately\nwhether the invariant is wrong or the code is. An invariant that is\nright but unprovable without a helper lemma produces the same message\nas one the code genuinely breaks, so the triage question is &ldquo;which of\nmy two claims is false?&rdquo; not &ldquo;how do I make the message stop.&rdquo;</p>\n<h2>Response playbook</h2>\n<p>When an invariant fails to discharge:</p>\n<ol>\n<li><strong>Read the counter-example first.</strong> It is the minimal violating\n   state, and it settles most triage in one look.</li>\n<li><strong>If the code is wrong, fix the code; if the invariant is wrong,\n   fix the statement.</strong> A proof of the wrong invariant is worse than\n   no proof, because it creates false confidence.</li>\n<li><strong>If both are right, supply the lemma.</strong> Proofs that fail for lack\n   of an intermediate step go through once the step is written down.</li>\n<li><strong>Demote only what cannot be proved.</strong> An invariant that resists\n   the prover can still run as a\n   <a class=\"body-link\" href=\"/sensors/runtime-invariants/\">runtime invariant</a> on every execution,\n   and it should.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Invariants that were never written down, and invariants whose statement is\nwrong. The gap between &ldquo;the invariant holds&rdquo; and &ldquo;the invariant is the one\nthe business needs&rdquo; is exactly what <a class=\"body-link\" href=\"/sensors/business-invariants/\">business invariants</a>\nmeasure from the other direction.</p>",
-      "body_text": "Invariants the compiler refuses to let you violate: Dafny invariant clauses, Frama-C annotations, JML specs, type-level witnesses like NonEmptyList . Where database invariants are checked by a live system against live data, these are proved once, at build time, for all possible executions. In practice The reading is the same as a type-checker error, raised against a property of the program instead of a type: src/Queue.hs:23:1: Error: Liquid Type Mismatch Invariant `size q >= 0` could not be proved Counter-example: `q` after `dequeue (mkQueue [])` Or, at the type level: a compiler that refuses head [] because the list’s type carries a proof of non-emptiness. Either way the verdict arrives at build time, once, for all possible executions, and a counter-example when the checker can find one. The counter-example is the part to read first: it is the smallest state that violates the invariant, and it usually says immediately whether the invariant is wrong or the code is. An invariant that is right but unprovable without a helper lemma produces the same message as one the code genuinely breaks, so the triage question is “which of my two claims is false?” not “how do I make the message stop.” Response playbook When an invariant fails to discharge: Read the counter-example first. It is the minimal violating state, and it settles most triage in one look. If the code is wrong, fix the code; if the invariant is wrong, fix the statement. A proof of the wrong invariant is worse than no proof, because it creates false confidence. If both are right, supply the lemma. Proofs that fail for lack of an intermediate step go through once the step is written down. Demote only what cannot be proved. An invariant that resists the prover can still run as a runtime invariant on every execution, and it should. What it cannot detect Invariants that were never written down, and invariants whose statement is wrong. The gap between “the invariant holds” and “the invariant is the one the business needs” is exactly what business invariants measure from the other direction.",
+      "body_html": "<p>Invariants the compiler refuses to let you violate: Dafny <code>invariant</code>\nclauses, Frama-C annotations, JML specs, type-level witnesses like\n<code>NonEmptyList</code>. Where <a class=\"body-link\" href=\"/sensors/database-invariants/\">database invariants</a> are\nchecked by a live system against live data, these are proved once, at build\ntime, for all possible executions.</p>\n<h2>In practice</h2>\n<p>The reading is the same as a <a class=\"body-link\" href=\"/sensors/type-checker/\">type-checker</a> error,\nraised against a property of the program instead of a type. Liquid\nHaskell prints the refinement it could infer next to the one the\nsignature demanded:</p>\n<pre><code>**** LIQUID: UNSAFE ************************************************************\n\nsrc/Queue.hs:23:34: error:\n    Liquid Type Mismatch\n    .\n    The inferred type\n      VV : {v : GHC.Types.Int | v == size q - 1}\n    .\n    is not a subtype of the required type\n      VV : {VV : GHC.Types.Int | VV &gt;= 0}\n    .\n    in the context\n      q : Queue a\n   |\n23 |   dequeue q = Q (tail (elems q)) (size q - 1)\n   |                                   ^^^^^^^^^^\n</code></pre>\n<p>Or, at the type level: a compiler that refuses <code>head []</code> because the\nlist&rsquo;s type carries a proof of non-emptiness. Either way the verdict\narrives at build time, once, for all possible executions.</p>\n<p>The two refinements are the reading, and the gap between them is the\nfinding: <code>size q - 1</code> is what the code establishes, <code>&gt;= 0</code> is what was\nclaimed, and nothing rules out <code>size q == 0</code>. Note what is <em>not</em> in the\nmessage — no failing input, no execution. An SMT-backed checker reports\nan unsatisfiable constraint, not a witness. Where a witness can be had at\nall it is a side feature and a weak one: Liquid Haskell&rsquo;s\n<code>--counter-examples</code> is flagged experimental, and Dafny&rsquo;s\n<code>--extract-counterexample</code> ships with the warning that it &ldquo;cannot\nguarantee that the counterexample it reports provably violates the\nassertion&rdquo;, and should be &ldquo;treated as a hint&rdquo;. Solver models are not\nminimal and are often unreachable. An invariant that is right but\nunprovable without a helper lemma produces the same message as one the\ncode genuinely breaks, so the triage question is &ldquo;which of my two claims\nis false?&rdquo; not &ldquo;how do I make the message stop.&rdquo;</p>\n<h2>Response playbook</h2>\n<p>When an invariant fails to discharge:</p>\n<ol>\n<li><strong>Read the inferred and required refinements against each other\n   first.</strong> The gap between what the code establishes and what the\n   signature demanded settles most triage in one look, and it is there\n   in every message — unlike a counter-example, which most of these\n   checkers will not give you.</li>\n<li><strong>If the code is wrong, fix the code; if the invariant is wrong,\n   fix the statement.</strong> A proof of the wrong invariant is worse than\n   no proof, because it creates false confidence.</li>\n<li><strong>If both are right, supply the lemma.</strong> Proofs that fail for lack\n   of an intermediate step go through once the step is written down.</li>\n<li><strong>Demote only what cannot be proved.</strong> An invariant that resists\n   the prover can still run as a\n   <a class=\"body-link\" href=\"/sensors/runtime-invariants/\">runtime invariant</a> on every execution,\n   and it should.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Invariants that were never written down, and invariants whose statement is\nwrong. The gap between &ldquo;the invariant holds&rdquo; and &ldquo;the invariant is the one\nthe business needs&rdquo; is exactly what <a class=\"body-link\" href=\"/sensors/business-invariants/\">business invariants</a>\nmeasure from the other direction.</p>",
+      "body_text": "Invariants the compiler refuses to let you violate: Dafny invariant clauses, Frama-C annotations, JML specs, type-level witnesses like NonEmptyList . Where database invariants are checked by a live system against live data, these are proved once, at build time, for all possible executions. In practice The reading is the same as a type-checker error, raised against a property of the program instead of a type. Liquid Haskell prints the refinement it could infer next to the one the signature demanded: **** LIQUID: UNSAFE ************************************************************ src/Queue.hs:23:34: error: Liquid Type Mismatch . The inferred type VV : {v : GHC.Types.Int | v == size q - 1} . is not a subtype of the required type VV : {VV : GHC.Types.Int | VV >= 0} . in the context q : Queue a | 23 | dequeue q = Q (tail (elems q)) (size q - 1) | ^^^^^^^^^^ Or, at the type level: a compiler that refuses head [] because the list’s type carries a proof of non-emptiness. Either way the verdict arrives at build time, once, for all possible executions. The two refinements are the reading, and the gap between them is the finding: size q - 1 is what the code establishes, >= 0 is what was claimed, and nothing rules out size q == 0 . Note what is not in the message — no failing input, no execution. An SMT-backed checker reports an unsatisfiable constraint, not a witness. Where a witness can be had at all it is a side feature and a weak one: Liquid Haskell’s --counter-examples is flagged experimental, and Dafny’s --extract-counterexample ships with the warning that it “cannot guarantee that the counterexample it reports provably violates the assertion”, and should be “treated as a hint”. Solver models are not minimal and are often unreachable. An invariant that is right but unprovable without a helper lemma produces the same message as one the code genuinely breaks, so the triage question is “which of my two claims is false?” not “how do I make the message stop.” Response playbook When an invariant fails to discharge: Read the inferred and required refinements against each other first. The gap between what the code establishes and what the signature demanded settles most triage in one look, and it is there in every message — unlike a counter-example, which most of these checkers will not give you. If the code is wrong, fix the code; if the invariant is wrong, fix the statement. A proof of the wrong invariant is worse than no proof, because it creates false confidence. If both are right, supply the lemma. Proofs that fail for lack of an intermediate step go through once the step is written down. Demote only what cannot be proved. An invariant that resists the prover can still run as a runtime invariant on every execution, and it should. What it cannot detect Invariants that were never written down, and invariants whose statement is wrong. The gap between “the invariant holds” and “the invariant is the one the business needs” is exactly what business invariants measure from the other direction.",
       "family": "invariants",
       "frontmatter": {
         "actionability": "blocking",
@@ -7852,6 +7884,11 @@
             "title": "Liquid Haskell",
             "url": "https://ucsd-progsys.github.io/liquidhaskell/"
           },
+          {
+            "kind": "other",
+            "title": "Dafny user guide, verification debugging",
+            "url": "https://dafny.org/latest/DafnyRef/DafnyRef#sec-counterexamples"
+          },
           {
             "description": "Static analysis and verification for C",
             "kind": "tool",
@@ -7874,20 +7911,22 @@
       "see_also_ids": [
         "SO-004",
         "SO-012",
-        "SO-001d"
+        "SO-001d",
+        "SO-017"
       ],
       "see_also_pages": [],
       "slug": "statically-checked-invariants",
       "title": "Statically Checked Invariants",
       "tokens": [
         "013",
-        "after",
         "against",
         "all",
         "and",
         "annotations",
         "are",
         "arrives",
+        "assertion",
+        "backed",
         "because",
         "between",
         "both",
@@ -7900,48 +7939,72 @@
         "carries",
         "checked",
         "checker",
+        "checkers",
+        "claimed",
         "claims",
         "clauses",
         "code",
         "compiler",
         "confidence",
+        "constraint",
+        "context",
         "could",
         "counter",
+        "counterexample",
         "creates",
         "dafny",
         "data",
         "database",
+        "demanded",
         "demote",
         "dequeue",
         "detect",
         "direction",
         "discharge",
         "down",
+        "each",
         "either",
+        "elems",
         "emptiness",
         "error",
+        "establishes",
         "every",
         "exactly",
         "example",
+        "examples",
         "execution",
         "executions",
+        "experimental",
+        "extract",
         "fail",
+        "failing",
         "fails",
         "false",
-        "find",
+        "feature",
+        "finding",
         "first",
         "fix",
+        "flagged",
         "for",
         "frama",
         "from",
         "gap",
         "genuinely",
+        "ghc",
+        "give",
+        "guarantee",
+        "had",
+        "haskell",
         "head",
         "helper",
+        "hint",
         "holds",
         "how",
-        "immediately",
+        "infer",
+        "inferred",
+        "input",
         "instead",
+        "int",
         "intermediate",
         "invariant",
         "invariants",
@@ -7960,26 +8023,32 @@
         "message",
         "minimal",
         "mismatch",
-        "mkqueue",
+        "models",
         "most",
         "needs",
         "never",
+        "next",
         "non",
         "nonemptylist",
         "not",
+        "note",
+        "nothing",
+        "often",
         "once",
         "one",
         "only",
         "other",
-        "part",
+        "out",
         "playbook",
         "possible",
         "practice",
+        "prints",
         "produces",
         "program",
         "proof",
         "proofs",
         "property",
+        "provably",
         "proved",
         "prover",
         "question",
@@ -7987,52 +8056,72 @@
         "raised",
         "read",
         "reading",
+        "refinement",
+        "refinements",
         "refuses",
+        "reports",
+        "required",
         "resists",
         "response",
         "right",
+        "rules",
         "run",
         "runtime",
         "same",
-        "says",
         "settles",
+        "ships",
         "should",
+        "side",
+        "signature",
         "size",
-        "smallest",
+        "smt",
+        "solver",
         "specs",
         "src",
-        "state",
         "statement",
         "statically",
         "step",
         "still",
         "stop",
+        "subtype",
         "supply",
         "system",
+        "tail",
         "than",
         "that",
         "the",
+        "them",
+        "there",
         "these",
         "through",
         "time",
+        "treated",
         "triage",
         "two",
         "type",
+        "types",
+        "unlike",
         "unprovable",
-        "usually",
+        "unreachable",
+        "unsafe",
+        "unsatisfiable",
         "verdict",
         "violate",
         "violates",
-        "violating",
+        "warning",
+        "was",
         "way",
+        "weak",
         "were",
         "what",
         "when",
         "where",
-        "whether",
         "which",
         "whose",
+        "will",
+        "with",
         "without",
+        "witness",
         "witnesses",
         "worse",
         "written",
@@ -8481,7 +8570,8 @@
       ],
       "see_also_ids": [
         "SO-004",
-        "SO-005"
+        "SO-005",
+        "SO-014b"
       ],
       "see_also_pages": [],
       "slug": "fault-injection",
@@ -8748,7 +8838,7 @@
       "url_path": "sensors/fault-injection/"
     },
     {
-      "body_html": "<p>What happens on inputs humans didn&rsquo;t think of? Fuzzing is a sensor of\n<em>robustness</em> against the infinite space of inputs the system will actually\nencounter — including inputs no engineer would ever write deliberately.</p>\n<h2>The oracle question</h2>\n<p>Property-based testing asks: &ldquo;does the implementation obey generalized\nproperties across huge input spaces?&rdquo; Fuzzing asks a simpler question: &ldquo;does\nit crash?&rdquo; The oracle is cheap — panics, exceptions, assertions, memory\nviolations — but the coverage of input space is enormous.</p>\n<pre><code># Coverage-guided fuzzing\n1. Generate random or mutated input\n2. Feed it to the system\n3. Did new code paths execute?\n   Yes → keep this input, mutate further\n   No  → discard, try again\n4. Did the system crash, panic, or violate an assertion?\n   Yes → save the input as a finding\n   No  → continue\n</code></pre>\n<blockquote>\n<p>Fuzzing is particularly powerful because it explores the input space that\nhumans systematically under-sample. An engineer writes tests for inputs\nthey can imagine. A fuzzer discovers inputs they can&rsquo;t.</p>\n</blockquote>\n<h2>In practice</h2>\n<p>A reading is a crash report: the sanitizer&rsquo;s verdict, the stack, and\nthe input that triggered it:</p>\n<pre><code>==4821==ERROR: AddressSanitizer: heap-buffer-overflow\n  on address 0x60200000f7f1 at pc 0x4b2f1b\nREAD of size 1 at 0x60200000f7f1\n    #0 parse_header   parser.c:142:9\n    #1 parse_message  parser.c:87:12\n    #2 main           harness.c:14:5\n\nartifact_prefix='./crashes/';\ntest unit written to ./crashes/crash-8f3a2c\n</code></pre>\n<p>Reading it well:</p>\n<ol>\n<li><strong>The saved input is the reading.</strong> The stack trace explains where\n   the system died; the file in <code>crashes/</code> is the evidence, and the\n   only part that reproduces the bug. Commit it before anything else.</li>\n<li><strong>Deduplicate by stack signature.</strong> Ten thousand crash files can\n   be one bug reached ten thousand ways. Grouping by the failing\n   frames turns a wall of red into a triage list.</li>\n<li><strong>No findings is a real reading, with conditions.</strong> A long\n   campaign that finds nothing says the explored input space is\n   robust, and it means more when the coverage counter was still\n   climbing. An empty report from a five-minute run means almost\n   nothing.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<ul>\n<li><strong>Exclude the crashing input.</strong> Adding the crash artifact to an\n  ignore list turns the finding into nothing; the bug stays live and\n  the report stays clean.</li>\n<li><strong>Run it too short to find.</strong> A five-minute campaign is a\n  checkbox. Empty reports from short runs are not evidence of\n  robustness, and campaign length is the lever being pulled.</li>\n<li><strong>Disable the oracle.</strong> Turning off the assertion or memory checks\n  that would have caught the crash removes the sensor while keeping\n  the run.</li>\n<li><strong>Mark crashes as theoretical.</strong> &ldquo;Nobody would send that input&rdquo; is\n  how fuzzing findings die. The input space the fuzzer explores is\n  exactly the one attackers and accidents sample.</li>\n</ul>\n<p>The meta-signal is the ignored-findings list. Anything on it is a\nlive bug the team chose to keep.</p>\n<h2>Response playbook</h2>\n<p>When the fuzzer finds a crash:</p>\n<ol>\n<li><strong>Commit the crashing input first.</strong> The file in the crashes\n   directory is the reading; everything else is commentary. It is\n   also the regression test for the fix.</li>\n<li><strong>Deduplicate by stack signature.</strong> Ten thousand crash files are\n   often one bug reached ten thousand ways. Group by the failing\n   frames before triaging.</li>\n<li><strong>Classify under the sanitizer.</strong> Memory error, assertion, or\n   panic decides severity: the first two are often exploitable or\n   corrupting, the last is a logic bug with a cheap fix.</li>\n<li><strong>Fix the root cause, not the site.</strong> Bounds-checking one\n   location while the parser is still wrong invites the fuzzer to\n   find the next location.</li>\n<li><strong>Re-run the campaign against the fix.</strong> The fix is confirmed\n   when the original input and its neighborhood stop crashing, not\n   when the code review lands.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Fuzzing with a crash oracle cannot detect <em>wrong but non-crashing</em> behavior.\nA function that returns the wrong answer without crashing will pass a fuzzer.\nFor correctness properties, pair fuzzing with <a class=\"body-link\" href=\"/sensors/metamorphic-testing/\">property-based\ntesting</a> or <a class=\"body-link\" href=\"/sensors/mutation-testing/\">mutation testing</a>.</p>",
+      "body_html": "<p>What happens on inputs humans didn&rsquo;t think of? Fuzzing is a sensor of\n<em>robustness</em> against the infinite space of inputs the system will actually\nencounter — including inputs no engineer would ever write deliberately.</p>\n<h2>The oracle question</h2>\n<p>Property-based testing asks: &ldquo;does the implementation obey generalized\nproperties across huge input spaces?&rdquo; Fuzzing asks a simpler question: &ldquo;does\nit crash?&rdquo; The oracle is cheap — panics, exceptions, assertions, memory\nviolations — but the coverage of input space is enormous.</p>\n<pre><code># Coverage-guided fuzzing\n1. Generate random or mutated input\n2. Feed it to the system\n3. Did new code paths execute?\n   Yes → keep this input, mutate further\n   No  → discard, try again\n4. Did the system crash, panic, or violate an assertion?\n   Yes → save the input as a finding\n   No  → continue\n</code></pre>\n<blockquote>\n<p>Fuzzing is particularly powerful because it explores the input space that\nhumans systematically under-sample. An engineer writes tests for inputs\nthey can imagine. A fuzzer discovers inputs they can&rsquo;t.</p>\n</blockquote>\n<h2>In practice</h2>\n<p>A reading is a crash report: the sanitizer&rsquo;s verdict, the stack, and\nthe input that triggered it:</p>\n<pre><code>==4821==ERROR: AddressSanitizer: heap-buffer-overflow\n  on address 0x60200000f7f1 at pc 0x4b2f1b\nREAD of size 1 at 0x60200000f7f1\n    #0 parse_header   parser.c:142:9\n    #1 parse_message  parser.c:87:12\n    #2 main           harness.c:14:5\n\nartifact_prefix='./crashes/';\ntest unit written to ./crashes/crash-8f3a2c\n</code></pre>\n<p>Reading it well:</p>\n<ol>\n<li><strong>The saved input is the reading.</strong> The stack trace explains where\n   the system died; the file in <code>crashes/</code> is the evidence, and the\n   only part that reproduces the bug. Commit it before anything else.</li>\n<li><strong>Deduplicate by stack signature.</strong> Ten thousand crash files can\n   be one bug reached ten thousand ways. Grouping by the failing\n   frames turns a wall of red into a triage list.</li>\n<li><strong>No findings is a real reading, with conditions.</strong> A long\n   campaign that finds nothing says the explored input space is\n   robust, and it means more when the coverage counter was still\n   climbing. An empty report from a five-minute run means almost\n   nothing.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<ul>\n<li><strong>Exclude the crashing input.</strong> Adding the crash artifact to an\n  ignore list turns the finding into nothing; the bug stays live and\n  the report stays clean.</li>\n<li><strong>Run it too short to find.</strong> A five-minute campaign is a\n  checkbox. Empty reports from short runs are not evidence of\n  robustness, and campaign length is the lever being pulled.</li>\n<li><strong>Disable the oracle.</strong> Turning off the assertion or memory checks\n  that would have caught the crash removes the sensor while keeping\n  the run.</li>\n<li><strong>Mark crashes as theoretical.</strong> &ldquo;Nobody would send that input&rdquo; is\n  how fuzzing findings die. The input space the fuzzer explores is\n  exactly the one attackers and accidents sample.</li>\n</ul>\n<p>The meta-signal is the ignored-findings list. Anything on it is a\nlive bug the team chose to keep.</p>\n<h2>Response playbook</h2>\n<p>When the fuzzer finds a crash:</p>\n<ol>\n<li><strong>Commit the crashing input first.</strong> The file in the crashes\n   directory is the reading; everything else is commentary. It is\n   also the regression test for the fix.</li>\n<li><strong>Deduplicate by stack signature.</strong> Ten thousand crash files are\n   often one bug reached ten thousand ways. Group by the failing\n   frames before triaging.</li>\n<li><strong>Classify under the sanitizer.</strong> Memory error, assertion, or\n   panic decides severity: the first two are often exploitable or\n   corrupting, the last is a logic bug with a cheap fix.</li>\n<li><strong>Fix the root cause, not the site.</strong> Bounds-checking one\n   location while the parser is still wrong invites the fuzzer to\n   find the next location.</li>\n<li><strong>Re-run the campaign against the fix.</strong> The fix is confirmed\n   when the original input and its neighborhood stop crashing, not\n   when the code review lands.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Fuzzing with a crash oracle cannot detect <em>wrong but non-crashing</em> behavior.\nA function that returns the wrong answer without crashing will pass a fuzzer.\nFor correctness properties, pair fuzzing with <a class=\"body-link\" href=\"/sensors/property-based-testing/\">property-based\ntesting</a> or <a class=\"body-link\" href=\"/sensors/mutation-testing/\">mutation testing</a>.</p>",
       "body_text": "What happens on inputs humans didn’t think of? Fuzzing is a sensor of robustness against the infinite space of inputs the system will actually encounter — including inputs no engineer would ever write deliberately. The oracle question Property-based testing asks: “does the implementation obey generalized properties across huge input spaces?” Fuzzing asks a simpler question: “does it crash?” The oracle is cheap — panics, exceptions, assertions, memory violations — but the coverage of input space is enormous. # Coverage-guided fuzzing 1. Generate random or mutated input 2. Feed it to the system 3. Did new code paths execute? Yes → keep this input, mutate further No → discard, try again 4. Did the system crash, panic, or violate an assertion? Yes → save the input as a finding No → continue Fuzzing is particularly powerful because it explores the input space that humans systematically under-sample. An engineer writes tests for inputs they can imagine. A fuzzer discovers inputs they can’t. In practice A reading is a crash report: the sanitizer’s verdict, the stack, and the input that triggered it: ==4821==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000f7f1 at pc 0x4b2f1b READ of size 1 at 0x60200000f7f1 #0 parse_header parser.c:142:9 #1 parse_message parser.c:87:12 #2 main harness.c:14:5 artifact_prefix='./crashes/'; test unit written to ./crashes/crash-8f3a2c Reading it well: The saved input is the reading. The stack trace explains where the system died; the file in crashes/ is the evidence, and the only part that reproduces the bug. Commit it before anything else. Deduplicate by stack signature. Ten thousand crash files can be one bug reached ten thousand ways. Grouping by the failing frames turns a wall of red into a triage list. No findings is a real reading, with conditions. A long campaign that finds nothing says the explored input space is robust, and it means more when the coverage counter was still climbing. An empty report from a five-minute run means almost nothing. How it gets gamed Exclude the crashing input. Adding the crash artifact to an ignore list turns the finding into nothing; the bug stays live and the report stays clean. Run it too short to find. A five-minute campaign is a checkbox. Empty reports from short runs are not evidence of robustness, and campaign length is the lever being pulled. Disable the oracle. Turning off the assertion or memory checks that would have caught the crash removes the sensor while keeping the run. Mark crashes as theoretical. “Nobody would send that input” is how fuzzing findings die. The input space the fuzzer explores is exactly the one attackers and accidents sample. The meta-signal is the ignored-findings list. Anything on it is a live bug the team chose to keep. Response playbook When the fuzzer finds a crash: Commit the crashing input first. The file in the crashes directory is the reading; everything else is commentary. It is also the regression test for the fix. Deduplicate by stack signature. Ten thousand crash files are often one bug reached ten thousand ways. Group by the failing frames before triaging. Classify under the sanitizer. Memory error, assertion, or panic decides severity: the first two are often exploitable or corrupting, the last is a logic bug with a cheap fix. Fix the root cause, not the site. Bounds-checking one location while the parser is still wrong invites the fuzzer to find the next location. Re-run the campaign against the fix. The fix is confirmed when the original input and its neighborhood stop crashing, not when the code review lands. What it cannot detect Fuzzing with a crash oracle cannot detect wrong but non-crashing behavior. A function that returns the wrong answer without crashing will pass a fuzzer. For correctness properties, pair fuzzing with property-based testing or mutation testing .",
       "family": "adversarial",
       "frontmatter": {
@@ -8827,7 +8917,8 @@
       ],
       "see_also_ids": [
         "SO-005b",
-        "SO-003"
+        "SO-003",
+        "SO-005e"
       ],
       "see_also_pages": [],
       "slug": "fuzzing",
@@ -9169,7 +9260,8 @@
       "see_also_ids": [
         "SO-005c",
         "SO-006c",
-        "SO-004b"
+        "SO-004b",
+        "SO-005d"
       ],
       "see_also_pages": [],
       "slug": "live-chaos-experiments",
@@ -9406,8 +9498,8 @@
       "url_path": "sensors/live-chaos-experiments/"
     },
     {
-      "body_html": "<p>You don&rsquo;t know the answer, but you know <em>how the answer should change</em>. This\nis a particularly beautiful sensor because you don&rsquo;t need an oracle.</p>\n<h2>Metamorphic relations</h2>\n<p>A metamorphic relation is a statement about how the output of a function\nshould change when the input changes in a specific way:</p>\n<pre><code>sort(sort(x))       == sort(x)         # idempotence\ndecrypt(encrypt(x)) == x               # round-trip\nsort(x) == sort(shuffle(x))            # order-independence\n\n# Numeric examples:\nabs(x) == abs(-x)                      # sign-independence\nabs(x + y) &lt;= abs(x) + abs(y)          # triangle inequality\nmax(x, y) == max(y, x)                 # commutativity\n</code></pre>\n<p>You don&rsquo;t need to know what <code>sort(x)</code> returns. You just need to know that\n<code>sort(sort(x))</code> should equal <code>sort(x)</code>. If it doesn&rsquo;t, something is wrong —\nand you&rsquo;ve found a bug without ever needing to compute the correct answer.</p>\n<blockquote>\n<p>Metamorphic testing is the answer to &ldquo;how do you test something when you\ncan&rsquo;t compute the expected output?&rdquo; Fuzzing explores the input space.\nMutation testing perturbs the implementation. Metamorphic testing perturbs\nthe input and checks relations between outputs. All three are adversarial\n— all three try to make the system fail.</p>\n</blockquote>\n<h2>In practice</h2>\n<p>A reading is a relation violation, shrunk to the smallest input that\nbreaks it:</p>\n<pre><code>______________ test_sort_is_idempotent ______________\n\nFalsifying example:\n    x = [2, 1, 3]\n\nassert sort(sort(x)) == sort(x)\nAssertionError: [1, 2, 3] != [1, 3, 2]\n\nShrunk to minimal failing input: x = [2, 1]\n</code></pre>\n<p>Reading it well:</p>\n<ol>\n<li><strong>The named relation is the oracle.</strong> The failure says which\n   relation broke (idempotence, round-trip, commutativity), which\n   tells you what kind of bug to look for before you look at the\n   code.</li>\n<li><strong>Work from the shrunk example.</strong> The original random input is\n   noise; the minimized case is the one you can verify by hand in\n   seconds. If you cannot hand-check it, shrink further.</li>\n<li><strong>A relation that never fails deserves a glance.</strong> It may be a\n   strong invariant, or it may be vacuous. Check that it would have\n   fired on a known-bad version of the code.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<ul>\n<li><strong>Weaken the relation.</strong> Replacing equality with &ldquo;same length,&rdquo; or\n  adding assumption filters, makes violations disappear by narrowing\n  what the relation claims. The test still runs and now detects\n  less.</li>\n<li><strong>Shrink the campaign.</strong> Cutting the example count until failures\n  stop appearing keeps the sensor&rsquo;s name and discards its reach.</li>\n<li><strong>Label violations as flaky.</strong> A relation that fails on one input\n  in a thousand is failing; retrying until it passes converts a\n  finding into noise.</li>\n</ul>\n<p>The meta-signal is the ratio of discarded (assumed-away) examples to\ngenerated ones. As it climbs, the relation is being strangled.</p>\n<h2>Response playbook</h2>\n<p>When a relation is violated:</p>\n<ol>\n<li><strong>Work from the shrunk example.</strong> The minimized input is the one\n   a human can verify by hand in seconds. If you cannot hand-check\n   it, shrink it further before debugging.</li>\n<li><strong>Decide which side of the relation is wrong.</strong> Usually the\n   implementation. Occasionally the relation overclaims, and then\n   the fix is a corrected relation, written with the reason, not a\n   deleted test.</li>\n<li><strong>Fix the implementation and re-run the campaign.</strong> A relation\n   violation is rarely one input wide; the same bug usually breaks a\n   neighborhood.</li>\n<li><strong>Pin the counterexample as a regression test.</strong> The shrunk input\n   is a free <a class=\"body-link\" href=\"/sensors/example-based-tests/\">example-based test</a> that\n   runs in milliseconds and guards the fix forever.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Metamorphic testing can only check relations you <em>know</em>. If a function has\nno obvious metamorphic relations, this sensor has nothing to test. It also\ncannot detect <a class=\"body-link\" href=\"/sensors/mutation-testing/\">missing behavior</a> — if a feature is\nabsent, there&rsquo;s no function to check relations on.</p>",
-      "body_text": "You don’t know the answer, but you know how the answer should change . This is a particularly beautiful sensor because you don’t need an oracle. Metamorphic relations A metamorphic relation is a statement about how the output of a function should change when the input changes in a specific way: sort(sort(x)) == sort(x) # idempotence decrypt(encrypt(x)) == x # round-trip sort(x) == sort(shuffle(x)) # order-independence # Numeric examples: abs(x) == abs(-x) # sign-independence abs(x + y) <= abs(x) + abs(y) # triangle inequality max(x, y) == max(y, x) # commutativity You don’t need to know what sort(x) returns. You just need to know that sort(sort(x)) should equal sort(x) . If it doesn’t, something is wrong — and you’ve found a bug without ever needing to compute the correct answer. Metamorphic testing is the answer to “how do you test something when you can’t compute the expected output?” Fuzzing explores the input space. Mutation testing perturbs the implementation. Metamorphic testing perturbs the input and checks relations between outputs. All three are adversarial — all three try to make the system fail. In practice A reading is a relation violation, shrunk to the smallest input that breaks it: ______________ test_sort_is_idempotent ______________ Falsifying example: x = [2, 1, 3] assert sort(sort(x)) == sort(x) AssertionError: [1, 2, 3] != [1, 3, 2] Shrunk to minimal failing input: x = [2, 1] Reading it well: The named relation is the oracle. The failure says which relation broke (idempotence, round-trip, commutativity), which tells you what kind of bug to look for before you look at the code. Work from the shrunk example. The original random input is noise; the minimized case is the one you can verify by hand in seconds. If you cannot hand-check it, shrink further. A relation that never fails deserves a glance. It may be a strong invariant, or it may be vacuous. Check that it would have fired on a known-bad version of the code. How it gets gamed Weaken the relation. Replacing equality with “same length,” or adding assumption filters, makes violations disappear by narrowing what the relation claims. The test still runs and now detects less. Shrink the campaign. Cutting the example count until failures stop appearing keeps the sensor’s name and discards its reach. Label violations as flaky. A relation that fails on one input in a thousand is failing; retrying until it passes converts a finding into noise. The meta-signal is the ratio of discarded (assumed-away) examples to generated ones. As it climbs, the relation is being strangled. Response playbook When a relation is violated: Work from the shrunk example. The minimized input is the one a human can verify by hand in seconds. If you cannot hand-check it, shrink it further before debugging. Decide which side of the relation is wrong. Usually the implementation. Occasionally the relation overclaims, and then the fix is a corrected relation, written with the reason, not a deleted test. Fix the implementation and re-run the campaign. A relation violation is rarely one input wide; the same bug usually breaks a neighborhood. Pin the counterexample as a regression test. The shrunk input is a free example-based test that runs in milliseconds and guards the fix forever. What it cannot detect Metamorphic testing can only check relations you know . If a function has no obvious metamorphic relations, this sensor has nothing to test. It also cannot detect missing behavior — if a feature is absent, there’s no function to check relations on.",
+      "body_html": "<p>You don&rsquo;t know the answer, but you know <em>how the answer should change</em>. This\nis a particularly beautiful sensor because you don&rsquo;t need an oracle.</p>\n<h2>Metamorphic relations</h2>\n<p>A metamorphic relation is a statement about how the output of a function\nshould change when the input changes in a specific way:</p>\n<pre><code>sort(sort(x))       == sort(x)         # idempotence\ndecrypt(encrypt(x)) == x               # round-trip\nsort(x) == sort(shuffle(x))            # order-independence\n\n# Numeric examples:\nabs(x) == abs(-x)                      # sign-independence\nabs(x + y) &lt;= abs(x) + abs(y)          # triangle inequality\nmax(x, y) == max(y, x)                 # commutativity\n</code></pre>\n<p>You don&rsquo;t need to know what <code>sort(x)</code> returns. You just need to know that\n<code>sort(sort(x))</code> should equal <code>sort(x)</code>. If it doesn&rsquo;t, something is wrong —\nand you&rsquo;ve found a bug without ever needing to compute the correct answer.</p>\n<blockquote>\n<p>Metamorphic testing is the answer to &ldquo;how do you test something when you\ncan&rsquo;t compute the expected output?&rdquo; Fuzzing explores the input space.\nMutation testing perturbs the implementation. Metamorphic testing perturbs\nthe input and checks relations between outputs. All three are adversarial\n— all three try to make the system fail.</p>\n</blockquote>\n<h2>In practice</h2>\n<p>A reading is a relation violation, already reduced. The relation below\nis order-independence — sorting a list and sorting its reverse must\nagree, which is checkable without knowing what the sorted answer is —\nrun against a <code>my_sort</code> that makes one bubble pass instead of a full\nsort:</p>\n<pre><code>xs = [0, 0, -1]\n\n    @given(st.lists(st.integers()))\n    def test_sort_is_order_independent(xs):\n&gt;       assert my_sort(xs) == my_sort(list(reversed(xs)))\nE       assert [0, -1, 0] == [-1, 0, 0]\nE         At index 0 diff: 0 != -1\nE         Use -v to get more diff\nE       Failing test case: test_sort_is_order_independent(\nE           xs=[0, 0, -1],\nE       )\n</code></pre>\n<p>Reading it well:</p>\n<ol>\n<li><strong>The named relation is the oracle.</strong> The failure says which\n   relation broke (idempotence, round-trip, commutativity), which\n   tells you what kind of bug to look for before you look at the\n   code.</li>\n<li><strong>The printed case is already the shrunk one.</strong> Hypothesis reports\n   the reduced input and nothing else; the larger inputs that also\n   failed never reach the console. If the printed case is still too\n   big to hand-check, the shrinker was blocked — usually by an\n   <code>assume</code> filter or a test that is not deterministic.</li>\n<li><strong>A relation that never fails deserves a glance.</strong> It may be a\n   strong invariant, or it may be vacuous. Check that it would have\n   fired on a known-bad version of the code.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<ul>\n<li><strong>Weaken the relation.</strong> Replacing equality with &ldquo;same length,&rdquo; or\n  adding assumption filters, makes violations disappear by narrowing\n  what the relation claims. The test still runs and now detects\n  less.</li>\n<li><strong>Shrink the campaign.</strong> Cutting the example count until failures\n  stop appearing keeps the sensor&rsquo;s name and discards its reach.</li>\n<li><strong>Label violations as flaky.</strong> A relation that fails on one input\n  in a thousand is failing; retrying until it passes converts a\n  finding into noise.</li>\n</ul>\n<p>The meta-signal is the ratio of discarded (assumed-away) examples to\ngenerated ones. As it climbs, the relation is being strangled.</p>\n<h2>Response playbook</h2>\n<p>When a relation is violated:</p>\n<ol>\n<li><strong>Work from the shrunk example.</strong> The minimized input is the one\n   a human can verify by hand in seconds. If you cannot hand-check\n   it, shrink it further before debugging.</li>\n<li><strong>Decide which side of the relation is wrong.</strong> Usually the\n   implementation. Occasionally the relation overclaims, and then\n   the fix is a corrected relation, written with the reason, not a\n   deleted test.</li>\n<li><strong>Fix the implementation and re-run the campaign.</strong> A relation\n   violation is rarely one input wide; the same bug usually breaks a\n   neighborhood.</li>\n<li><strong>Pin the counterexample as a regression test.</strong> The shrunk input\n   is a free <a class=\"body-link\" href=\"/sensors/example-based-tests/\">example-based test</a> that\n   runs in milliseconds and guards the fix forever.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Metamorphic testing can only check relations you <em>know</em>. If a function has\nno obvious metamorphic relations, this sensor has nothing to test. It also\ncannot detect <a class=\"body-link\" href=\"/sensors/mutation-testing/\">missing behavior</a> — if a feature is\nabsent, there&rsquo;s no function to check relations on.</p>",
+      "body_text": "You don’t know the answer, but you know how the answer should change . This is a particularly beautiful sensor because you don’t need an oracle. Metamorphic relations A metamorphic relation is a statement about how the output of a function should change when the input changes in a specific way: sort(sort(x)) == sort(x) # idempotence decrypt(encrypt(x)) == x # round-trip sort(x) == sort(shuffle(x)) # order-independence # Numeric examples: abs(x) == abs(-x) # sign-independence abs(x + y) <= abs(x) + abs(y) # triangle inequality max(x, y) == max(y, x) # commutativity You don’t need to know what sort(x) returns. You just need to know that sort(sort(x)) should equal sort(x) . If it doesn’t, something is wrong — and you’ve found a bug without ever needing to compute the correct answer. Metamorphic testing is the answer to “how do you test something when you can’t compute the expected output?” Fuzzing explores the input space. Mutation testing perturbs the implementation. Metamorphic testing perturbs the input and checks relations between outputs. All three are adversarial — all three try to make the system fail. In practice A reading is a relation violation, already reduced. The relation below is order-independence — sorting a list and sorting its reverse must agree, which is checkable without knowing what the sorted answer is — run against a my_sort that makes one bubble pass instead of a full sort: xs = [0, 0, -1] @given(st.lists(st.integers())) def test_sort_is_order_independent(xs): > assert my_sort(xs) == my_sort(list(reversed(xs))) E assert [0, -1, 0] == [-1, 0, 0] E At index 0 diff: 0 != -1 E Use -v to get more diff E Failing test case: test_sort_is_order_independent( E xs=[0, 0, -1], E ) Reading it well: The named relation is the oracle. The failure says which relation broke (idempotence, round-trip, commutativity), which tells you what kind of bug to look for before you look at the code. The printed case is already the shrunk one. Hypothesis reports the reduced input and nothing else; the larger inputs that also failed never reach the console. If the printed case is still too big to hand-check, the shrinker was blocked — usually by an assume filter or a test that is not deterministic. A relation that never fails deserves a glance. It may be a strong invariant, or it may be vacuous. Check that it would have fired on a known-bad version of the code. How it gets gamed Weaken the relation. Replacing equality with “same length,” or adding assumption filters, makes violations disappear by narrowing what the relation claims. The test still runs and now detects less. Shrink the campaign. Cutting the example count until failures stop appearing keeps the sensor’s name and discards its reach. Label violations as flaky. A relation that fails on one input in a thousand is failing; retrying until it passes converts a finding into noise. The meta-signal is the ratio of discarded (assumed-away) examples to generated ones. As it climbs, the relation is being strangled. Response playbook When a relation is violated: Work from the shrunk example. The minimized input is the one a human can verify by hand in seconds. If you cannot hand-check it, shrink it further before debugging. Decide which side of the relation is wrong. Usually the implementation. Occasionally the relation overclaims, and then the fix is a corrected relation, written with the reason, not a deleted test. Fix the implementation and re-run the campaign. A relation violation is rarely one input wide; the same bug usually breaks a neighborhood. Pin the counterexample as a regression test. The shrunk input is a free example-based test that runs in milliseconds and guards the fix forever. What it cannot detect Metamorphic testing can only check relations you know . If a function has no obvious metamorphic relations, this sensor has nothing to test. It also cannot detect missing behavior — if a feature is absent, there’s no function to check relations on.",
       "family": "adversarial",
       "frontmatter": {
         "actionability": "guiding",
@@ -9477,27 +9569,30 @@
       ],
       "see_also_ids": [
         "SO-005",
-        "SO-003"
+        "SO-003",
+        "SO-005e"
       ],
       "see_also_pages": [],
       "slug": "metamorphic-testing",
       "title": "Metamorphic Testing",
       "tokens": [
         "005b",
-        "______________",
         "about",
         "abs",
         "absent",
         "adding",
         "adversarial",
+        "against",
+        "agree",
         "all",
+        "already",
         "also",
         "and",
         "answer",
         "appearing",
         "are",
         "assert",
-        "assertionerror",
+        "assume",
         "assumed",
         "assumption",
         "away",
@@ -9508,9 +9603,13 @@
         "before",
         "behavior",
         "being",
+        "below",
         "between",
+        "big",
+        "blocked",
         "breaks",
         "broke",
+        "bubble",
         "bug",
         "but",
         "campaign",
@@ -9520,12 +9619,14 @@
         "change",
         "changes",
         "check",
+        "checkable",
         "checks",
         "claims",
         "climbs",
         "code",
         "commutativity",
         "compute",
+        "console",
         "converts",
         "correct",
         "corrected",
@@ -9535,15 +9636,19 @@
         "debugging",
         "decide",
         "decrypt",
+        "def",
         "deleted",
         "deserves",
         "detect",
         "detects",
+        "deterministic",
+        "diff",
         "disappear",
         "discarded",
         "discards",
         "doesn",
         "don",
+        "else",
         "encrypt",
         "equal",
         "equality",
@@ -9553,12 +9658,13 @@
         "expected",
         "explores",
         "fail",
+        "failed",
         "failing",
         "fails",
         "failure",
         "failures",
-        "falsifying",
         "feature",
+        "filter",
         "filters",
         "finding",
         "fired",
@@ -9569,12 +9675,15 @@
         "found",
         "free",
         "from",
+        "full",
         "function",
         "further",
         "fuzzing",
         "gamed",
         "generated",
+        "get",
         "gets",
+        "given",
         "glance",
         "guards",
         "hand",
@@ -9582,11 +9691,16 @@
         "have",
         "how",
         "human",
+        "hypothesis",
         "idempotence",
         "implementation",
         "independence",
+        "index",
         "inequality",
         "input",
+        "inputs",
+        "instead",
+        "integers",
         "into",
         "invariant",
         "its",
@@ -9594,10 +9708,14 @@
         "keeps",
         "kind",
         "know",
+        "knowing",
         "known",
         "label",
+        "larger",
         "length",
         "less",
+        "list",
+        "lists",
         "look",
         "make",
         "makes",
@@ -9606,10 +9724,12 @@
         "meta",
         "metamorphic",
         "milliseconds",
-        "minimal",
         "minimized",
         "missing",
+        "more",
+        "must",
         "mutation",
+        "my_sort",
         "name",
         "named",
         "narrowing",
@@ -9629,29 +9749,33 @@
         "only",
         "oracle",
         "order",
-        "original",
         "output",
         "outputs",
         "overclaims",
         "particularly",
+        "pass",
         "passes",
         "perturbs",
         "pin",
         "playbook",
         "practice",
-        "random",
+        "printed",
         "rarely",
         "ratio",
         "reach",
         "reading",
         "reason",
+        "reduced",
         "regression",
         "relation",
         "relations",
         "replacing",
+        "reports",
         "response",
         "retrying",
         "returns",
+        "reverse",
+        "reversed",
         "round",
         "run",
         "runs",
@@ -9661,14 +9785,16 @@
         "sensor",
         "should",
         "shrink",
+        "shrinker",
         "shrunk",
         "shuffle",
         "side",
         "sign",
         "signal",
-        "smallest",
         "something",
         "sort",
+        "sorted",
+        "sorting",
         "space",
         "specific",
         "statement",
@@ -9679,7 +9805,7 @@
         "system",
         "tells",
         "test",
-        "test_sort_is_idempotent",
+        "test_sort_is_order_independent",
         "testing",
         "that",
         "the",
@@ -9688,10 +9814,12 @@
         "this",
         "thousand",
         "three",
+        "too",
         "triangle",
         "trip",
         "try",
         "until",
+        "use",
         "usually",
         "vacuous",
         "verify",
@@ -9699,6 +9827,7 @@
         "violated",
         "violation",
         "violations",
+        "was",
         "way",
         "weaken",
         "well",
@@ -9717,8 +9846,8 @@
       "url_path": "sensors/metamorphic-testing/"
     },
     {
-      "body_html": "<p>You state a property that should hold for <em>every</em> input, and the tool\ngenerates inputs trying to break it. Where <a class=\"body-link\" href=\"/sensors/example-based-tests/\">example-based\ntesting</a> checks one case you thought of,\nproperty-based testing searches the space of cases you didn&rsquo;t.</p>\n<h2>Properties vs examples</h2>\n<p>A property is a universal statement:</p>\n<pre><code class=\"language-python\">@given(st.lists(st.integers()))\ndef test_sort_idempotent(xs):\n    assert sort(xs) == sort(sort(xs))\n</code></pre>\n<p>A failing run is a counterexample, shrunk to the smallest input that\nstill breaks it:</p>\n<pre><code>Falsifying example: test_sort_idempotent(xs=[1, 0])\nShrunk to: xs=[1, 0]\nassert sort([1, 0]) == sort(sort([1, 0]))\nAssertionError: [0, 1] != [1, 0]\n</code></pre>\n<p>The shrink is the actionable part — it turns a random failure into a\ncase you can verify by hand in seconds.</p>\n<h2>How it differs from metamorphic testing</h2>\n<p>Property-based testing and <a class=\"body-link\" href=\"/sensors/metamorphic-testing/\">metamorphic testing</a>\nare siblings in the adversarial family. Property-based testing states a\nproperty directly (<code>f(x) == f(-x)</code>, <code>sort(sort(xs)) == sort(xs)</code>) and\nrelies on the generator to find an <code>x</code> that breaks it. Metamorphic\ntesting states a <em>relation between outputs</em> — you don&rsquo;t know the\nanswer, only how the answer should change when the input changes. In\npractice most property-based test suites contain metamorphic relations\n(idempotence, commutativity, round-trip) and the distinction is mostly\nabout whether you can name the property outright or only the relation\nbetween two calls.</p>\n<h2>How it gets gamed</h2>\n<ul>\n<li><strong>Weaken the property.</strong> Replacing equality with &ldquo;same length,&rdquo; or\n  adding <code>assume</code> filters that discard failing inputs, makes the\n  property hold while detecting less. The test still runs.</li>\n<li><strong>Shrink the campaign.</strong> Cutting the example count until failures\n  stop appearing keeps the sensor&rsquo;s name and discards its reach.</li>\n<li><strong>Label violations as flaky.</strong> A property that fails on one input in\n  a thousand is failing; retrying until it passes converts a finding\n  into noise.</li>\n</ul>\n<p>The meta-signal is the ratio of discarded (assumed-away) examples to\ngenerated ones. As it climbs, the property is being strangled.</p>\n<h2>Response playbook</h2>\n<p>When a property fails:</p>\n<ol>\n<li><strong>Work from the shrunk example.</strong> The minimized input is the one a\n   human can verify by hand in seconds. If you can&rsquo;t hand-check it,\n   shrink further before debugging.</li>\n<li><strong>Decide which side is wrong.</strong> Usually the implementation.\n   Occasionally the property overclaims, and the fix is a corrected\n   property, written with the reason, not a deleted test.</li>\n<li><strong>Fix the implementation and re-run the campaign.</strong> A property\n   violation is rarely one input wide; the same bug usually breaks a\n   neighborhood.</li>\n<li><strong>Pin the counterexample as a regression test.</strong> The shrunk input\n   is a free <a class=\"body-link\" href=\"/sensors/example-based-tests/\">example-based test</a> that runs\n   in milliseconds and guards the fix forever.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Property-based testing can only check properties you <em>state</em>. If a\nfunction has no obvious property, the generator has nothing to falsify.\nIt also cannot detect <a class=\"body-link\" href=\"/sensors/mutation-testing/\">missing behavior</a> — if a\nfeature is absent, there&rsquo;s no property to check against it. And unlike\n<a class=\"body-link\" href=\"/sensors/fuzzing/\">fuzzing</a>, it assumes you can characterize correctness;\nfuzzing finds crashes even when you can&rsquo;t write down a property.</p>",
-      "body_text": "You state a property that should hold for every input, and the tool generates inputs trying to break it. Where example-based testing checks one case you thought of, property-based testing searches the space of cases you didn’t. Properties vs examples A property is a universal statement: @given(st.lists(st.integers())) def test_sort_idempotent(xs): assert sort(xs) == sort(sort(xs)) A failing run is a counterexample, shrunk to the smallest input that still breaks it: Falsifying example: test_sort_idempotent(xs=[1, 0]) Shrunk to: xs=[1, 0] assert sort([1, 0]) == sort(sort([1, 0])) AssertionError: [0, 1] != [1, 0] The shrink is the actionable part — it turns a random failure into a case you can verify by hand in seconds. How it differs from metamorphic testing Property-based testing and metamorphic testing are siblings in the adversarial family. Property-based testing states a property directly ( f(x) == f(-x) , sort(sort(xs)) == sort(xs) ) and relies on the generator to find an x that breaks it. Metamorphic testing states a relation between outputs — you don’t know the answer, only how the answer should change when the input changes. In practice most property-based test suites contain metamorphic relations (idempotence, commutativity, round-trip) and the distinction is mostly about whether you can name the property outright or only the relation between two calls. How it gets gamed Weaken the property. Replacing equality with “same length,” or adding assume filters that discard failing inputs, makes the property hold while detecting less. The test still runs. Shrink the campaign. Cutting the example count until failures stop appearing keeps the sensor’s name and discards its reach. Label violations as flaky. A property that fails on one input in a thousand is failing; retrying until it passes converts a finding into noise. The meta-signal is the ratio of discarded (assumed-away) examples to generated ones. As it climbs, the property is being strangled. Response playbook When a property fails: Work from the shrunk example. The minimized input is the one a human can verify by hand in seconds. If you can’t hand-check it, shrink further before debugging. Decide which side is wrong. Usually the implementation. Occasionally the property overclaims, and the fix is a corrected property, written with the reason, not a deleted test. Fix the implementation and re-run the campaign. A property violation is rarely one input wide; the same bug usually breaks a neighborhood. Pin the counterexample as a regression test. The shrunk input is a free example-based test that runs in milliseconds and guards the fix forever. What it cannot detect Property-based testing can only check properties you state . If a function has no obvious property, the generator has nothing to falsify. It also cannot detect missing behavior — if a feature is absent, there’s no property to check against it. And unlike fuzzing , it assumes you can characterize correctness; fuzzing finds crashes even when you can’t write down a property.",
+      "body_html": "<p>You state a property that should hold for <em>every</em> input, and the tool\ngenerates inputs trying to break it. Where <a class=\"body-link\" href=\"/sensors/example-based-tests/\">example-based\ntesting</a> checks one case you thought of,\nproperty-based testing searches the space of cases you didn&rsquo;t.</p>\n<h2>Properties vs examples</h2>\n<p>A property is a universal statement. Here it is stated against a\n<code>my_sort</code> that orders numbers lexicographically — <code>sorted(xs, key=str)</code>,\na bug that hides behind any example whose inputs are non-negative single\ndigits:</p>\n<pre><code class=\"language-python\">@given(st.lists(st.integers()))\ndef test_sort_is_ordered(xs):\n    ys = my_sort(xs)\n    assert ys == sorted(ys)\n</code></pre>\n<p>The reading is one falsifying input, already shrunk. Hypothesis runs the\nproperty until it fails, reduces the failure, and reports only the\nreduced case — never the input that first broke it:</p>\n<pre><code>xs = [-1, -2]\n\n    @given(st.lists(st.integers()))\n    def test_sort_is_ordered(xs):\n        ys = my_sort(xs)\n&gt;       assert ys == sorted(ys)\nE       assert [-1, -2] == [-2, -1]\nE         At index 0 diff: -1 != -2\nE         Use -v to get more diff\nE       Failing test case: test_sort_is_ordered(\nE           xs=[-1, -2],\nE       )\n</code></pre>\n<p>Two elements and a minus sign, verifiable by hand in seconds: that\nreduction is the actionable part of the sensor, and it is invisible in\nthe output: the larger inputs that also failed are never printed. (The\nbanner is version-dependent — Hypothesis 6 prints\n<code>Failing test case:</code> where older releases printed <code>Falsifying\nexample:</code> — so grep for the parameter line, not the banner.)</p>\n<h2>How it differs from metamorphic testing</h2>\n<p>Property-based testing and <a class=\"body-link\" href=\"/sensors/metamorphic-testing/\">metamorphic testing</a>\nare siblings in the adversarial family. Property-based testing states a\nproperty directly (<code>f(x) == f(-x)</code>, <code>sort(sort(xs)) == sort(xs)</code>) and\nrelies on the generator to find an <code>x</code> that breaks it. Metamorphic\ntesting states a <em>relation between outputs</em> — you don&rsquo;t know the\nanswer, only how the answer should change when the input changes. In\npractice most property-based test suites contain metamorphic relations\n(idempotence, commutativity, round-trip) and the distinction is mostly\nabout whether you can name the property outright or only the relation\nbetween two calls.</p>\n<h2>How it gets gamed</h2>\n<ul>\n<li><strong>Weaken the property.</strong> Replacing equality with &ldquo;same length,&rdquo; or\n  adding <code>assume</code> filters that discard failing inputs, makes the\n  property hold while detecting less. The test still runs.</li>\n<li><strong>Shrink the campaign.</strong> Cutting the example count until failures\n  stop appearing keeps the sensor&rsquo;s name and discards its reach.</li>\n<li><strong>Label violations as flaky.</strong> A property that fails on one input in\n  a thousand is failing; retrying until it passes converts a finding\n  into noise.</li>\n</ul>\n<p>The meta-signal is the ratio of discarded (assumed-away) examples to\ngenerated ones. As it climbs, the property is being strangled.</p>\n<h2>Response playbook</h2>\n<p>When a property fails:</p>\n<ol>\n<li><strong>Work from the shrunk example.</strong> The minimized input is the one a\n   human can verify by hand in seconds. If you can&rsquo;t hand-check it,\n   shrink further before debugging.</li>\n<li><strong>Decide which side is wrong.</strong> Usually the implementation.\n   Occasionally the property overclaims, and the fix is a corrected\n   property, written with the reason, not a deleted test.</li>\n<li><strong>Fix the implementation and re-run the campaign.</strong> A property\n   violation is rarely one input wide; the same bug usually breaks a\n   neighborhood.</li>\n<li><strong>Pin the counterexample as a regression test.</strong> The shrunk input\n   is a free <a class=\"body-link\" href=\"/sensors/example-based-tests/\">example-based test</a> that runs\n   in milliseconds and guards the fix forever.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>Property-based testing can only check properties you <em>state</em>. If a\nfunction has no obvious property, the generator has nothing to falsify.\nIt also cannot detect <a class=\"body-link\" href=\"/sensors/mutation-testing/\">missing behavior</a> — if a\nfeature is absent, there&rsquo;s no property to check against it. And unlike\n<a class=\"body-link\" href=\"/sensors/fuzzing/\">fuzzing</a>, it assumes you can characterize correctness;\nfuzzing finds crashes even when you can&rsquo;t write down a property.</p>",
+      "body_text": "You state a property that should hold for every input, and the tool generates inputs trying to break it. Where example-based testing checks one case you thought of, property-based testing searches the space of cases you didn’t. Properties vs examples A property is a universal statement. Here it is stated against a my_sort that orders numbers lexicographically — sorted(xs, key=str) , a bug that hides behind any example whose inputs are non-negative single digits: @given(st.lists(st.integers())) def test_sort_is_ordered(xs): ys = my_sort(xs) assert ys == sorted(ys) The reading is one falsifying input, already shrunk. Hypothesis runs the property until it fails, reduces the failure, and reports only the reduced case — never the input that first broke it: xs = [-1, -2] @given(st.lists(st.integers())) def test_sort_is_ordered(xs): ys = my_sort(xs) > assert ys == sorted(ys) E assert [-1, -2] == [-2, -1] E At index 0 diff: -1 != -2 E Use -v to get more diff E Failing test case: test_sort_is_ordered( E xs=[-1, -2], E ) Two elements and a minus sign, verifiable by hand in seconds: that reduction is the actionable part of the sensor, and it is invisible in the output: the larger inputs that also failed are never printed. (The banner is version-dependent — Hypothesis 6 prints Failing test case: where older releases printed Falsifying example: — so grep for the parameter line, not the banner.) How it differs from metamorphic testing Property-based testing and metamorphic testing are siblings in the adversarial family. Property-based testing states a property directly ( f(x) == f(-x) , sort(sort(xs)) == sort(xs) ) and relies on the generator to find an x that breaks it. Metamorphic testing states a relation between outputs — you don’t know the answer, only how the answer should change when the input changes. In practice most property-based test suites contain metamorphic relations (idempotence, commutativity, round-trip) and the distinction is mostly about whether you can name the property outright or only the relation between two calls. How it gets gamed Weaken the property. Replacing equality with “same length,” or adding assume filters that discard failing inputs, makes the property hold while detecting less. The test still runs. Shrink the campaign. Cutting the example count until failures stop appearing keeps the sensor’s name and discards its reach. Label violations as flaky. A property that fails on one input in a thousand is failing; retrying until it passes converts a finding into noise. The meta-signal is the ratio of discarded (assumed-away) examples to generated ones. As it climbs, the property is being strangled. Response playbook When a property fails: Work from the shrunk example. The minimized input is the one a human can verify by hand in seconds. If you can’t hand-check it, shrink further before debugging. Decide which side is wrong. Usually the implementation. Occasionally the property overclaims, and the fix is a corrected property, written with the reason, not a deleted test. Fix the implementation and re-run the campaign. A property violation is rarely one input wide; the same bug usually breaks a neighborhood. Pin the counterexample as a regression test. The shrunk input is a free example-based test that runs in milliseconds and guards the fix forever. What it cannot detect Property-based testing can only check properties you state . If a function has no obvious property, the generator has nothing to falsify. It also cannot detect missing behavior — if a feature is absent, there’s no property to check against it. And unlike fuzzing , it assumes you can characterize correctness; fuzzing finds crashes even when you can’t write down a property.",
       "family": "adversarial",
       "frontmatter": {
         "actionability": "guiding",
@@ -9790,7 +9919,8 @@
       "see_also_ids": [
         "SO-005",
         "SO-005b",
-        "SO-003"
+        "SO-003",
+        "SO-002b"
       ],
       "see_also_pages": [],
       "slug": "property-based-testing",
@@ -9803,24 +9933,28 @@
         "adding",
         "adversarial",
         "against",
+        "already",
         "also",
         "and",
         "answer",
+        "any",
         "appearing",
         "are",
         "assert",
-        "assertionerror",
         "assume",
         "assumed",
         "assumes",
         "away",
+        "banner",
         "based",
         "before",
         "behavior",
+        "behind",
         "being",
         "between",
         "break",
         "breaks",
+        "broke",
         "bug",
         "calls",
         "campaign",
@@ -9847,10 +9981,13 @@
         "decide",
         "def",
         "deleted",
+        "dependent",
         "detect",
         "detecting",
         "didn",
+        "diff",
         "differs",
+        "digits",
         "directly",
         "discard",
         "discarded",
@@ -9858,11 +9995,13 @@
         "distinction",
         "don",
         "down",
+        "elements",
         "equality",
         "even",
         "every",
         "example",
         "examples",
+        "failed",
         "failing",
         "fails",
         "failure",
@@ -9875,6 +10014,7 @@
         "find",
         "finding",
         "finds",
+        "first",
         "fix",
         "flaky",
         "for",
@@ -9888,65 +10028,94 @@
         "generated",
         "generates",
         "generator",
+        "get",
         "gets",
         "given",
+        "grep",
         "guards",
         "hand",
         "has",
+        "here",
+        "hides",
         "hold",
         "how",
         "human",
+        "hypothesis",
         "idempotence",
         "implementation",
+        "index",
         "input",
         "inputs",
         "integers",
         "into",
+        "invisible",
         "its",
         "keeps",
+        "key",
         "know",
         "label",
+        "larger",
         "length",
         "less",
+        "lexicographically",
+        "line",
         "lists",
         "makes",
         "meta",
         "metamorphic",
         "milliseconds",
         "minimized",
+        "minus",
         "missing",
+        "more",
         "most",
         "mostly",
+        "my_sort",
         "name",
+        "negative",
         "neighborhood",
+        "never",
         "noise",
+        "non",
         "not",
         "nothing",
+        "numbers",
         "obvious",
         "occasionally",
+        "older",
         "one",
         "ones",
         "only",
+        "orders",
+        "output",
         "outputs",
         "outright",
         "overclaims",
+        "parameter",
         "part",
         "passes",
         "pin",
         "playbook",
         "practice",
+        "printed",
+        "prints",
         "properties",
         "property",
-        "random",
         "rarely",
         "ratio",
         "reach",
+        "reading",
         "reason",
+        "reduced",
+        "reduces",
+        "reduction",
         "regression",
         "relation",
         "relations",
+        "releases",
         "relies",
         "replacing",
+        "reports",
         "response",
         "retrying",
         "round",
@@ -9961,19 +10130,23 @@
         "shrunk",
         "siblings",
         "side",
+        "sign",
         "signal",
-        "smallest",
+        "single",
         "sort",
+        "sorted",
         "space",
         "state",
+        "stated",
         "statement",
         "states",
         "still",
         "stop",
+        "str",
         "strangled",
         "suites",
         "test",
-        "test_sort_idempotent",
+        "test_sort_is_ordered",
         "testing",
         "that",
         "the",
@@ -9983,13 +10156,15 @@
         "tool",
         "trip",
         "trying",
-        "turns",
         "two",
         "universal",
         "unlike",
         "until",
+        "use",
         "usually",
+        "verifiable",
         "verify",
+        "version",
         "violation",
         "violations",
         "weaken",
@@ -9999,6 +10174,7 @@
         "whether",
         "which",
         "while",
+        "whose",
         "wide",
         "with",
         "work",
@@ -10010,8 +10186,8 @@
       "url_path": "sensors/property-based-testing/"
     },
     {
-      "body_html": "<p>Attacking the code before it runs. Taint tracking, dataflow analysis, and\npattern-based scanners (Semgrep, CodeQL) ask: &ldquo;is there any path through\nthis program where an adversary&rsquo;s input reaches a dangerous sink?&rdquo; A sensor\nof exploitable structure, not of known-bad strings.</p>\n<h2>In practice</h2>\n<p>A scanner reading is a source-to-sink path with a severity attached,\nnot a bare pattern match:</p>\n<pre><code>semgrep --config p/owasp-top-ten src/\nsrc/api/search.py:31: Possible SQL injection (high)\n  query = &quot;SELECT * FROM items WHERE name = '%s'&quot; % request.args[&quot;q&quot;]\n  taint: request.args[&quot;q&quot;] -&gt; cursor.execute(query)\n</code></pre>\n<p>The value is the path. The finding is not &ldquo;this string looks\ndangerous&rdquo; but &ldquo;input from here reaches that sink through these\nlines,&rdquo; and the reading takes minutes because the intermediate steps\nare printed.</p>\n<p>Severity is the scanner&rsquo;s estimate, not the exploitability. A high\nfinding behind an authenticated, sanitized code path may matter less\nthan a medium one on an anonymous endpoint, and the scanner knows\nneither the auth model nor the traffic. Which is why triage reads the\npath first and the badge second, and why the same finding can be a\nblocker in one service and a note in another.</p>\n<h2>Response playbook</h2>\n<p>When the scanner fires:</p>\n<ol>\n<li><strong>Trace the full path before judging severity.</strong> The scanner shows\n   the chain; confirm whether anything in between already\n   sanitizes, authenticates, or bounds the input.</li>\n<li><strong>If reachable, fix at the sink.</strong> Parameterized queries,\n   escaping APIs, allowlists. Point fixes outlive pattern fixes.</li>\n<li><strong>If unreachable, record why.</strong> A one-line justification at the\n   finding site keeps the next reviewer from redoing the triage.</li>\n<li><strong>Fix the class, then add the rule.</strong> If the sink exists in ten\n   other places, the scanner&rsquo;s job is to find all ten; wire that\n   rule into CI as blocking before the next one ships.</li>\n<li><strong>Treat untriaged findings as broken builds.</strong> A backlog of\n   undated security findings is a queue that never drains.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<ul>\n<li><strong>Severity downgrade by triage.</strong> Marking a finding &ldquo;won&rsquo;t fix&rdquo;\n  with a one-word justification is the cheapest override available\n  to a reviewer. The meta-signal is the ratio of suppressed to\n  confirmed findings, and the age of the suppressed ones.</li>\n<li><strong>Baseline erosion.</strong> Freezing the finding count in a baseline\n  file and only failing on new ones, while the baseline quietly\n  grows forever because &ldquo;old findings are someone else&rsquo;s problem.&rdquo;</li>\n<li><strong>Rule-set narrowing.</strong> Disabling taint rules because they are\n  noisy and keeping only the pattern matches, which detect the\n  least of the three.</li>\n<li><strong>Scanner shopping.</strong> Running the tool that finds the fewest\n  issues in review and calling it &ldquo;our SAST.&rdquo;</li>\n</ul>\n<h2>What it cannot detect</h2>\n<p>Vulnerabilities in the composition of services, in configuration, or in the\ndependencies&rsquo; runtime behavior — those belong to\n<a class=\"body-link\" href=\"/sensors/fault-injection/\">fault injection</a> and live chaos. And a clean scan\nsays nothing about attacks that arrive through valid inputs:\n<a class=\"body-link\" href=\"/sensors/fuzzing/\">fuzzing</a> covers that side.</p>\n<p>Security is a deeper topic than this catalog covers. SAST is one sensor in\nthe adversarial family — it asks whether an adversary&rsquo;s input can reach a\ndangerous sink — but the broader practice of application security\n(threat modeling, penetration testing, dependency vulnerabilities, runtime\nexploit detection) deserves its own resources. See\n<a class=\"body-link\" href=\"https://owasp.org\">OWASP</a> and the\n<a class=\"body-link\" href=\"https://cwe.mitre.org\">CWE</a> for dedicated treatment.</p>",
-      "body_text": "Attacking the code before it runs. Taint tracking, dataflow analysis, and pattern-based scanners (Semgrep, CodeQL) ask: “is there any path through this program where an adversary’s input reaches a dangerous sink?” A sensor of exploitable structure, not of known-bad strings. In practice A scanner reading is a source-to-sink path with a severity attached, not a bare pattern match: semgrep --config p/owasp-top-ten src/ src/api/search.py:31: Possible SQL injection (high) query = \"SELECT * FROM items WHERE name = '%s'\" % request.args[\"q\"] taint: request.args[\"q\"] -> cursor.execute(query) The value is the path. The finding is not “this string looks dangerous” but “input from here reaches that sink through these lines,” and the reading takes minutes because the intermediate steps are printed. Severity is the scanner’s estimate, not the exploitability. A high finding behind an authenticated, sanitized code path may matter less than a medium one on an anonymous endpoint, and the scanner knows neither the auth model nor the traffic. Which is why triage reads the path first and the badge second, and why the same finding can be a blocker in one service and a note in another. Response playbook When the scanner fires: Trace the full path before judging severity. The scanner shows the chain; confirm whether anything in between already sanitizes, authenticates, or bounds the input. If reachable, fix at the sink. Parameterized queries, escaping APIs, allowlists. Point fixes outlive pattern fixes. If unreachable, record why. A one-line justification at the finding site keeps the next reviewer from redoing the triage. Fix the class, then add the rule. If the sink exists in ten other places, the scanner’s job is to find all ten; wire that rule into CI as blocking before the next one ships. Treat untriaged findings as broken builds. A backlog of undated security findings is a queue that never drains. How it gets gamed Severity downgrade by triage. Marking a finding “won’t fix” with a one-word justification is the cheapest override available to a reviewer. The meta-signal is the ratio of suppressed to confirmed findings, and the age of the suppressed ones. Baseline erosion. Freezing the finding count in a baseline file and only failing on new ones, while the baseline quietly grows forever because “old findings are someone else’s problem.” Rule-set narrowing. Disabling taint rules because they are noisy and keeping only the pattern matches, which detect the least of the three. Scanner shopping. Running the tool that finds the fewest issues in review and calling it “our SAST.” What it cannot detect Vulnerabilities in the composition of services, in configuration, or in the dependencies’ runtime behavior — those belong to fault injection and live chaos. And a clean scan says nothing about attacks that arrive through valid inputs: fuzzing covers that side. Security is a deeper topic than this catalog covers. SAST is one sensor in the adversarial family — it asks whether an adversary’s input can reach a dangerous sink — but the broader practice of application security (threat modeling, penetration testing, dependency vulnerabilities, runtime exploit detection) deserves its own resources. See OWASP and the CWE for dedicated treatment.",
+      "body_html": "<p>Attacking the code before it runs. Taint tracking, dataflow analysis, and\npattern-based scanners (Semgrep, CodeQL) ask: &ldquo;is there any path through\nthis program where an adversary&rsquo;s input reaches a dangerous sink?&rdquo; A sensor\nof exploitable structure, not of known-bad strings.</p>\n<h2>In practice</h2>\n<p>A scanner reading is a source-to-sink path with a severity attached,\nnot a bare pattern match:</p>\n<pre><code>semgrep --config p/owasp-top-ten src/\nsrc/api/search.py:31: Possible SQL injection (high)\n  query = &quot;SELECT * FROM items WHERE name = '%s'&quot; % request.args[&quot;q&quot;]\n  taint: request.args[&quot;q&quot;] -&gt; cursor.execute(query)\n</code></pre>\n<p>The value is the path. The finding is not &ldquo;this string looks\ndangerous&rdquo; but &ldquo;input from here reaches that sink through these\nlines,&rdquo; and the reading takes minutes because the intermediate steps\nare printed.</p>\n<p>Severity is the scanner&rsquo;s estimate, not the exploitability. A high\nfinding behind an authenticated, sanitized code path may matter less\nthan a medium one on an anonymous endpoint, and the scanner knows\nneither the auth model nor the traffic. Which is why triage reads the\npath first and the badge second, and why the same finding can be a\nblocker in one service and a note in another.</p>\n<h2>Response playbook</h2>\n<p>When the scanner fires:</p>\n<ol>\n<li><strong>Trace the full path before judging severity.</strong> The scanner shows\n   the chain; confirm whether anything in between already\n   sanitizes, authenticates, or bounds the input.</li>\n<li><strong>If reachable, fix at the sink.</strong> Parameterized queries,\n   escaping APIs, allowlists. Point fixes outlive pattern fixes.</li>\n<li><strong>If unreachable, record why.</strong> A one-line justification at the\n   finding site keeps the next reviewer from redoing the triage.</li>\n<li><strong>Fix the class, then add the rule.</strong> If the sink exists in ten\n   other places, the scanner&rsquo;s job is to find all ten; wire that\n   rule into CI as blocking before the next one ships.</li>\n<li><strong>Treat untriaged findings as broken builds.</strong> A backlog of\n   undated security findings is a queue that never drains.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<ul>\n<li><strong>Severity downgrade by triage.</strong> Marking a finding &ldquo;won&rsquo;t fix&rdquo;\n  with a one-word justification is the cheapest override available\n  to a reviewer. The meta-signal is the ratio of suppressed to\n  confirmed findings, and the age of the suppressed ones.</li>\n<li><strong>Baseline erosion.</strong> Freezing the finding count in a baseline\n  file and only failing on new ones, while the baseline quietly\n  grows forever because &ldquo;old findings are someone else&rsquo;s problem.&rdquo;</li>\n<li><strong>Rule-set narrowing.</strong> Disabling taint rules because they are\n  noisy and keeping only the pattern matches, which detect the\n  least of the three.</li>\n<li><strong>Scanner shopping.</strong> Running the tool that finds the fewest\n  issues in review and calling it &ldquo;our SAST.&rdquo;</li>\n</ul>\n<h2>What it cannot detect</h2>\n<p>Vulnerabilities in the composition of services, in configuration, or in the\ndependencies&rsquo; runtime behavior — those belong to\n<a class=\"body-link\" href=\"/sensors/fault-injection/\">fault injection</a> and <a class=\"body-link\" href=\"/sensors/live-chaos-experiments/\">live\nchaos</a>. And a clean scan says nothing about\nattacks that arrive through valid inputs: <a class=\"body-link\" href=\"/sensors/fuzzing/\">fuzzing</a> covers\nthat side.</p>\n<p>Security is a deeper topic than this catalog covers. SAST is one sensor in\nthe adversarial family — it asks whether an adversary&rsquo;s input can reach a\ndangerous sink — but the broader practice of application security\n(threat modeling, penetration testing, dependency vulnerabilities, runtime\nexploit detection) deserves its own resources. See\n<a class=\"body-link\" href=\"https://owasp.org\">OWASP</a> and the\n<a class=\"body-link\" href=\"https://cwe.mitre.org\">CWE</a> for dedicated treatment.</p>",
+      "body_text": "Attacking the code before it runs. Taint tracking, dataflow analysis, and pattern-based scanners (Semgrep, CodeQL) ask: “is there any path through this program where an adversary’s input reaches a dangerous sink?” A sensor of exploitable structure, not of known-bad strings. In practice A scanner reading is a source-to-sink path with a severity attached, not a bare pattern match: semgrep --config p/owasp-top-ten src/ src/api/search.py:31: Possible SQL injection (high) query = \"SELECT * FROM items WHERE name = '%s'\" % request.args[\"q\"] taint: request.args[\"q\"] -> cursor.execute(query) The value is the path. The finding is not “this string looks dangerous” but “input from here reaches that sink through these lines,” and the reading takes minutes because the intermediate steps are printed. Severity is the scanner’s estimate, not the exploitability. A high finding behind an authenticated, sanitized code path may matter less than a medium one on an anonymous endpoint, and the scanner knows neither the auth model nor the traffic. Which is why triage reads the path first and the badge second, and why the same finding can be a blocker in one service and a note in another. Response playbook When the scanner fires: Trace the full path before judging severity. The scanner shows the chain; confirm whether anything in between already sanitizes, authenticates, or bounds the input. If reachable, fix at the sink. Parameterized queries, escaping APIs, allowlists. Point fixes outlive pattern fixes. If unreachable, record why. A one-line justification at the finding site keeps the next reviewer from redoing the triage. Fix the class, then add the rule. If the sink exists in ten other places, the scanner’s job is to find all ten; wire that rule into CI as blocking before the next one ships. Treat untriaged findings as broken builds. A backlog of undated security findings is a queue that never drains. How it gets gamed Severity downgrade by triage. Marking a finding “won’t fix” with a one-word justification is the cheapest override available to a reviewer. The meta-signal is the ratio of suppressed to confirmed findings, and the age of the suppressed ones. Baseline erosion. Freezing the finding count in a baseline file and only failing on new ones, while the baseline quietly grows forever because “old findings are someone else’s problem.” Rule-set narrowing. Disabling taint rules because they are noisy and keeping only the pattern matches, which detect the least of the three. Scanner shopping. Running the tool that finds the fewest issues in review and calling it “our SAST.” What it cannot detect Vulnerabilities in the composition of services, in configuration, or in the dependencies’ runtime behavior — those belong to fault injection and live chaos . And a clean scan says nothing about attacks that arrive through valid inputs: fuzzing covers that side. Security is a deeper topic than this catalog covers. SAST is one sensor in the adversarial family — it asks whether an adversary’s input can reach a dangerous sink — but the broader practice of application security (threat modeling, penetration testing, dependency vulnerabilities, runtime exploit detection) deserves its own resources. See OWASP and the CWE for dedicated treatment.",
       "family": "adversarial",
       "frontmatter": {
         "actionability": "blocking",
@@ -10079,7 +10255,8 @@
       "see_also_ids": [
         "SO-005",
         "SO-005c",
-        "SO-001c"
+        "SO-001c",
+        "SO-014b"
       ],
       "see_also_pages": [],
       "slug": "static-security-analysis",
@@ -11416,7 +11593,8 @@
         "comprehension"
       ],
       "see_also_ids": [
-        "SO-004"
+        "SO-004",
+        "SO-006b"
       ],
       "see_also_pages": [],
       "slug": "observability-events",
@@ -11808,7 +11986,8 @@
       "see_also_ids": [
         "SO-006",
         "SO-006b",
-        "SO-006c"
+        "SO-006c",
+        "SO-006e"
       ],
       "see_also_pages": [],
       "slug": "resource-telemetry",
@@ -12173,7 +12352,8 @@
         "SO-007",
         "SO-007b",
         "SO-015",
-        "SO-009"
+        "SO-009",
+        "SO-012d"
       ],
       "see_also_pages": [],
       "slug": "ab-testing",
@@ -12840,7 +13020,8 @@
         "SO-007b",
         "SO-007c",
         "SO-007d",
-        "SO-005c"
+        "SO-005c",
+        "SO-007e"
       ],
       "see_also_pages": [],
       "slug": "canary-analysis",
@@ -13419,7 +13600,8 @@
       "see_also_ids": [
         "SO-007",
         "SO-006c",
-        "SO-012c"
+        "SO-012c",
+        "SO-007e"
       ],
       "see_also_pages": [],
       "slug": "feature-flag-exposure",
@@ -14336,7 +14518,8 @@
       "see_also_ids": [
         "SO-008",
         "SO-008b",
-        "SO-002"
+        "SO-002",
+        "SO-015c"
       ],
       "see_also_pages": [],
       "slug": "boundary-sensors",
@@ -14655,7 +14838,9 @@
       "see_also_ids": [
         "SO-008b",
         "SO-008c",
-        "SO-008d"
+        "SO-008d",
+        "SO-015c",
+        "SO-014d"
       ],
       "see_also_pages": [],
       "slug": "dependency-graph",
@@ -15266,7 +15451,8 @@
       "see_also_ids": [
         "SO-008",
         "SO-008b",
-        "SO-008d"
+        "SO-008d",
+        "SO-012d"
       ],
       "see_also_pages": [],
       "slug": "hotspot-analysis",
@@ -15603,7 +15789,8 @@
       "see_also_ids": [
         "SO-008b",
         "SO-006c",
-        "SO-006"
+        "SO-006",
+        "SO-008"
       ],
       "see_also_pages": [],
       "slug": "live-service-graph",
@@ -16157,8 +16344,8 @@
       "url_path": "sensors/change-coupling/"
     },
     {
-      "body_html": "<p>Deployment frequency, lead time for changes, change failure rate,\nreliability, time to restore. Five numbers about how changes have\nhistorically flowed through this organization — a sensor of delivery\n<em>pattern</em>, answering &ldquo;does our recent past look like teams that ship\nsafely?&rdquo; Reliability was added in DORA 2023 as the fifth metric, reflecting\nthat stability is measured by whether the system meets its reliability\ntargets, not just by how fast failures are fixed.</p>\n<h2>In practice</h2>\n<p>The reading is a dashboard of the five numbers over a trailing window,\ncompared against the team&rsquo;s own history rather than an industry table:</p>\n<table>\n<thead>\n<tr>\n<th>Metric</th>\n<th>Last quarter</th>\n<th>This quarter</th>\n<th>Trend</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Deploy frequency</td>\n<td>14 / week</td>\n<td>6 / week</td>\n<td>down</td>\n</tr>\n<tr>\n<td>Lead time for changes</td>\n<td>36 hours</td>\n<td>61 hours</td>\n<td>up (worse)</td>\n</tr>\n<tr>\n<td>Change failure rate</td>\n<td>4%</td>\n<td>11%</td>\n<td>up (worse)</td>\n</tr>\n<tr>\n<td>Time to restore</td>\n<td>45 minutes</td>\n<td>3 hours</td>\n<td>up (worse)</td>\n</tr>\n<tr>\n<td>Reliability target met</td>\n<td>yes</td>\n<td>no</td>\n<td>worse</td>\n</tr>\n</tbody>\n</table>\n<p>Reading it well:</p>\n<ol>\n<li><strong>Read the five together.</strong> Deployment frequency falling alone might\n   be a deliberate pause; falling alongside rising failure rate and\n   lead time is a pipeline in trouble. One number is an anecdote, five\n   are a pattern.</li>\n<li><strong>Compare against the team&rsquo;s own baseline.</strong> Industry quartiles\n   tell you what other teams do, not what changed here. The trend line\n   is the signal.</li>\n<li><strong>Distrust flat perfection.</strong> Metrics that never move usually mean\n   the instrumentation is stale or the definitions quietly widened.</li>\n<li><strong>Follow a bad number to its records.</strong> The dashboard locates the\n   quarter; the deploy log and incident timeline explain it.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<p>DORA metrics are classically gameable, because every one of them can be\nmoved without moving the underlying delivery health:</p>\n<ul>\n<li><strong>Batch deploys to inflate frequency.</strong> Shipping twenty small commits\n  as one &ldquo;deploy&rdquo; or splitting one change into ten trivial ones; the\n  frequency number moves either way, depending on which direction\n  flatters the dashboard.</li>\n<li><strong>Cherry-pick easy deploys.</strong> Counting only the low-risk service\n  whose pipeline is mature, while the painful path everyone actually\n  uses goes unmeasured.</li>\n<li><strong>Close incidents early.</strong> Marking an incident resolved when the page\n  stops, not when users are unaffected, buys time-to-restore without\n  restoring anything.</li>\n<li><strong>Redefine failure downward.</strong> If only rollbacks count as failures,\n  teams hotfix forward and the change failure rate falls by definition.</li>\n<li><strong>Tighten the reliability denominator.</strong> Meeting a target is easy\n  when the target was quietly negotiated down after the quarter\n  started.</li>\n</ul>\n<p>The meta-signal is definitional drift: log the metric definitions and\nthe counting rules alongside the numbers, and treat any unannounced\nchange to them as a finding in itself.</p>\n<h2>Response playbook</h2>\n<p>When the dashboard degrades:</p>\n<ol>\n<li><strong>Verify the definitions first.</strong> Before acting, confirm the numbers\n   were computed the same way as last quarter. Half of all &ldquo;sudden&rdquo;\n   DORA regressions are a pipeline or counting change, not a delivery\n   change.</li>\n<li><strong>Trace the lead-time increase stage by stage.</strong> Split lead time\n   into review, build, test, and deploy waits; the longest stage is\n   the one to attack, and it is almost never the one people guess.</li>\n<li><strong>Correlate failure rate with incident records.</strong> Use <a class=\"body-link\" href=\"/sensors/incident-correlation/\">incident\n   correlation</a> to see whether failures are\n   clustering in one service or one change type before prescribing a\n   fix.</li>\n<li><strong>Lower batch size, not ambition.</strong> If deploys are failing more,\n   ship smaller changes with <a class=\"body-link\" href=\"/sensors/canary-analysis/\">canary analysis</a>\n   in front of them; do not add another approval gate, which raises\n   lead time and failure rate together.</li>\n<li><strong>Re-measure after one quarter.</strong> The dashboard lags; give any fix a\n   full window before judging it, and say so when the numbers are\n   reported.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>The cause of a bad number. DORA metrics locate the problem in time and\nstage; understanding it requires <a class=\"body-link\" href=\"/sensors/decision-provenance/\">decision provenance</a>\nand <a class=\"body-link\" href=\"/sensors/incident-correlation/\">incident correlation</a>. And like all metrics\nthat become targets, they invite gaming — deployment frequency can be raised\nby shipping trivia.</p>",
-      "body_text": "Deployment frequency, lead time for changes, change failure rate, reliability, time to restore. Five numbers about how changes have historically flowed through this organization — a sensor of delivery pattern , answering “does our recent past look like teams that ship safely?” Reliability was added in DORA 2023 as the fifth metric, reflecting that stability is measured by whether the system meets its reliability targets, not just by how fast failures are fixed. In practice The reading is a dashboard of the five numbers over a trailing window, compared against the team’s own history rather than an industry table: Metric Last quarter This quarter Trend Deploy frequency 14 / week 6 / week down Lead time for changes 36 hours 61 hours up (worse) Change failure rate 4% 11% up (worse) Time to restore 45 minutes 3 hours up (worse) Reliability target met yes no worse Reading it well: Read the five together. Deployment frequency falling alone might be a deliberate pause; falling alongside rising failure rate and lead time is a pipeline in trouble. One number is an anecdote, five are a pattern. Compare against the team’s own baseline. Industry quartiles tell you what other teams do, not what changed here. The trend line is the signal. Distrust flat perfection. Metrics that never move usually mean the instrumentation is stale or the definitions quietly widened. Follow a bad number to its records. The dashboard locates the quarter; the deploy log and incident timeline explain it. How it gets gamed DORA metrics are classically gameable, because every one of them can be moved without moving the underlying delivery health: Batch deploys to inflate frequency. Shipping twenty small commits as one “deploy” or splitting one change into ten trivial ones; the frequency number moves either way, depending on which direction flatters the dashboard. Cherry-pick easy deploys. Counting only the low-risk service whose pipeline is mature, while the painful path everyone actually uses goes unmeasured. Close incidents early. Marking an incident resolved when the page stops, not when users are unaffected, buys time-to-restore without restoring anything. Redefine failure downward. If only rollbacks count as failures, teams hotfix forward and the change failure rate falls by definition. Tighten the reliability denominator. Meeting a target is easy when the target was quietly negotiated down after the quarter started. The meta-signal is definitional drift: log the metric definitions and the counting rules alongside the numbers, and treat any unannounced change to them as a finding in itself. Response playbook When the dashboard degrades: Verify the definitions first. Before acting, confirm the numbers were computed the same way as last quarter. Half of all “sudden” DORA regressions are a pipeline or counting change, not a delivery change. Trace the lead-time increase stage by stage. Split lead time into review, build, test, and deploy waits; the longest stage is the one to attack, and it is almost never the one people guess. Correlate failure rate with incident records. Use incident correlation to see whether failures are clustering in one service or one change type before prescribing a fix. Lower batch size, not ambition. If deploys are failing more, ship smaller changes with canary analysis in front of them; do not add another approval gate, which raises lead time and failure rate together. Re-measure after one quarter. The dashboard lags; give any fix a full window before judging it, and say so when the numbers are reported. What it cannot detect The cause of a bad number. DORA metrics locate the problem in time and stage; understanding it requires decision provenance and incident correlation . And like all metrics that become targets, they invite gaming — deployment frequency can be raised by shipping trivia.",
+      "body_html": "<p>Change lead time, deployment frequency, failed deployment recovery time,\nchange fail rate, deployment rework rate. Five numbers about how changes\nhave historically flowed through this organization — a sensor of delivery\n<em>pattern</em>, answering &ldquo;does our recent past look like teams that ship\nsafely?&rdquo; This entry describes the five-metric model DORA has published\nsince 2024: the first three grouped as throughput, the last two as\ninstability.</p>\n<p>The set is versioned, and the names move. &ldquo;Time to restore service&rdquo; was\nrenamed <em>failed deployment recovery time</em> in 2023, narrowing it to failures\na deployment caused rather than any outage; <em>deployment rework rate</em> arrived\nin 2024 as the actual fifth metric. Reliability was added in 2021 and is not\none of the five — DORA&rsquo;s own history records that the report calling it &ldquo;the\nfifth metric&rdquo; was inaccurate, and files it under operational rather than\ndelivery performance. A dashboard labelled &ldquo;the DORA metrics&rdquo; is dated by\nwhich names it uses, and comparing readings across a rename compares two\nsensors.</p>\n<h2>In practice</h2>\n<p>The reading is a dashboard of the five numbers over a trailing window,\ncompared against the team&rsquo;s own history rather than an industry table:</p>\n<table>\n<thead>\n<tr>\n<th>Metric</th>\n<th>Last quarter</th>\n<th>This quarter</th>\n<th>Trend</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Change lead time</td>\n<td>36 hours</td>\n<td>61 hours</td>\n<td>up (worse)</td>\n</tr>\n<tr>\n<td>Deployment frequency</td>\n<td>14 / week</td>\n<td>6 / week</td>\n<td>down</td>\n</tr>\n<tr>\n<td>Failed deployment recovery time</td>\n<td>45 minutes</td>\n<td>3 hours</td>\n<td>up (worse)</td>\n</tr>\n<tr>\n<td>Change fail rate</td>\n<td>4%</td>\n<td>11%</td>\n<td>up (worse)</td>\n</tr>\n<tr>\n<td>Deployment rework rate</td>\n<td>6%</td>\n<td>14%</td>\n<td>up (worse)</td>\n</tr>\n</tbody>\n</table>\n<p>Reading it well:</p>\n<ol>\n<li><strong>Read the five together.</strong> Deployment frequency falling alone might\n   be a deliberate pause; falling alongside rising failure rate and\n   lead time is a pipeline in trouble. One number is an anecdote, five\n   are a pattern.</li>\n<li><strong>Compare against the team&rsquo;s own baseline.</strong> Industry quartiles\n   tell you what other teams do, not what changed here. The trend line\n   is the signal.</li>\n<li><strong>Distrust flat perfection.</strong> Metrics that never move usually mean\n   the instrumentation is stale or the definitions quietly widened.</li>\n<li><strong>Follow a bad number to its records.</strong> The dashboard locates the\n   quarter; the deploy log and incident timeline explain it.</li>\n</ol>\n<h2>How it gets gamed</h2>\n<p>DORA metrics are classically gameable, because every one of them can be\nmoved without moving the underlying delivery health:</p>\n<ul>\n<li><strong>Batch deploys to inflate frequency.</strong> Shipping twenty small commits\n  as one &ldquo;deploy&rdquo; or splitting one change into ten trivial ones; the\n  frequency number moves either way, depending on which direction\n  flatters the dashboard.</li>\n<li><strong>Cherry-pick easy deploys.</strong> Counting only the low-risk service\n  whose pipeline is mature, while the painful path everyone actually\n  uses goes unmeasured.</li>\n<li><strong>Close incidents early.</strong> Marking an incident resolved when the page\n  stops, not when users are unaffected, buys recovery time without\n  restoring anything.</li>\n<li><strong>Redefine failure downward.</strong> If only rollbacks count as failures,\n  teams hotfix forward and the change failure rate falls by definition.</li>\n<li><strong>File rework as planned work.</strong> Deployment rework rate counts the\n  deployments that exist only to fix something already shipped. Attach\n  the hotfix to the next feature ticket and it stops counting, while\n  the rework goes on happening.</li>\n</ul>\n<p>The meta-signal is definitional drift: log the metric definitions and\nthe counting rules alongside the numbers, and treat any unannounced\nchange to them as a finding in itself.</p>\n<h2>Response playbook</h2>\n<p>When the dashboard degrades:</p>\n<ol>\n<li><strong>Verify the definitions first.</strong> Before acting, confirm the numbers\n   were computed the same way as last quarter. Half of all &ldquo;sudden&rdquo;\n   DORA regressions are a pipeline or counting change, not a delivery\n   change.</li>\n<li><strong>Trace the lead-time increase stage by stage.</strong> Split lead time\n   into review, build, test, and deploy waits; the longest stage is\n   the one to attack, and it is almost never the one people guess.</li>\n<li><strong>Correlate failure rate with incident records.</strong> Use <a class=\"body-link\" href=\"/sensors/incident-correlation/\">incident\n   correlation</a> to see whether failures are\n   clustering in one service or one change type before prescribing a\n   fix.</li>\n<li><strong>Lower batch size, not ambition.</strong> If deploys are failing more,\n   ship smaller changes with <a class=\"body-link\" href=\"/sensors/canary-analysis/\">canary analysis</a>\n   in front of them; do not add another approval gate, which raises\n   lead time and failure rate together.</li>\n<li><strong>Re-measure after one quarter.</strong> The dashboard lags; give any fix a\n   full window before judging it, and say so when the numbers are\n   reported.</li>\n</ol>\n<h2>What it cannot detect</h2>\n<p>The cause of a bad number. DORA metrics locate the problem in time and\nstage; understanding it requires <a class=\"body-link\" href=\"/sensors/decision-provenance/\">decision provenance</a>\nand <a class=\"body-link\" href=\"/sensors/incident-correlation/\">incident correlation</a>. And like all metrics\nthat become targets, they invite gaming — deployment frequency can be raised\nby shipping trivia.</p>",
+      "body_text": "Change lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate. Five numbers about how changes have historically flowed through this organization — a sensor of delivery pattern , answering “does our recent past look like teams that ship safely?” This entry describes the five-metric model DORA has published since 2024: the first three grouped as throughput, the last two as instability. The set is versioned, and the names move. “Time to restore service” was renamed failed deployment recovery time in 2023, narrowing it to failures a deployment caused rather than any outage; deployment rework rate arrived in 2024 as the actual fifth metric. Reliability was added in 2021 and is not one of the five — DORA’s own history records that the report calling it “the fifth metric” was inaccurate, and files it under operational rather than delivery performance. A dashboard labelled “the DORA metrics” is dated by which names it uses, and comparing readings across a rename compares two sensors. In practice The reading is a dashboard of the five numbers over a trailing window, compared against the team’s own history rather than an industry table: Metric Last quarter This quarter Trend Change lead time 36 hours 61 hours up (worse) Deployment frequency 14 / week 6 / week down Failed deployment recovery time 45 minutes 3 hours up (worse) Change fail rate 4% 11% up (worse) Deployment rework rate 6% 14% up (worse) Reading it well: Read the five together. Deployment frequency falling alone might be a deliberate pause; falling alongside rising failure rate and lead time is a pipeline in trouble. One number is an anecdote, five are a pattern. Compare against the team’s own baseline. Industry quartiles tell you what other teams do, not what changed here. The trend line is the signal. Distrust flat perfection. Metrics that never move usually mean the instrumentation is stale or the definitions quietly widened. Follow a bad number to its records. The dashboard locates the quarter; the deploy log and incident timeline explain it. How it gets gamed DORA metrics are classically gameable, because every one of them can be moved without moving the underlying delivery health: Batch deploys to inflate frequency. Shipping twenty small commits as one “deploy” or splitting one change into ten trivial ones; the frequency number moves either way, depending on which direction flatters the dashboard. Cherry-pick easy deploys. Counting only the low-risk service whose pipeline is mature, while the painful path everyone actually uses goes unmeasured. Close incidents early. Marking an incident resolved when the page stops, not when users are unaffected, buys recovery time without restoring anything. Redefine failure downward. If only rollbacks count as failures, teams hotfix forward and the change failure rate falls by definition. File rework as planned work. Deployment rework rate counts the deployments that exist only to fix something already shipped. Attach the hotfix to the next feature ticket and it stops counting, while the rework goes on happening. The meta-signal is definitional drift: log the metric definitions and the counting rules alongside the numbers, and treat any unannounced change to them as a finding in itself. Response playbook When the dashboard degrades: Verify the definitions first. Before acting, confirm the numbers were computed the same way as last quarter. Half of all “sudden” DORA regressions are a pipeline or counting change, not a delivery change. Trace the lead-time increase stage by stage. Split lead time into review, build, test, and deploy waits; the longest stage is the one to attack, and it is almost never the one people guess. Correlate failure rate with incident records. Use incident correlation to see whether failures are clustering in one service or one change type before prescribing a fix. Lower batch size, not ambition. If deploys are failing more, ship smaller changes with canary analysis in front of them; do not add another approval gate, which raises lead time and failure rate together. Re-measure after one quarter. The dashboard lags; give any fix a full window before judging it, and say so when the numbers are reported. What it cannot detect The cause of a bad number. DORA metrics locate the problem in time and stage; understanding it requires decision provenance and incident correlation . And like all metrics that become targets, they invite gaming — deployment frequency can be raised by shipping trivia.",
       "family": "evolution",
       "frontmatter": {
         "actionability": "guiding",
@@ -16192,6 +16379,11 @@
             "venue": "Puppet + DORA",
             "year": 2017
           },
+          {
+            "kind": "other",
+            "title": "A history of DORA's software delivery metrics",
+            "url": "https://dora.dev/insights/dora-metrics-history/"
+          },
           {
             "kind": "tool",
             "title": "DORA",
@@ -16211,7 +16403,7 @@
             "url": "https://dora.dev"
           },
           {
-            "description": "DORA's four-metric assessment tool",
+            "description": "DORA's research assessment tool",
             "kind": "tool",
             "title": "DevOps Research Assessment",
             "url": "https://dora.dev/research/"
@@ -16234,9 +16426,13 @@
       "title": "DORA Metrics",
       "tokens": [
         "015b",
+        "2021",
         "2023",
+        "2024",
         "about",
+        "across",
         "acting",
+        "actual",
         "actually",
         "add",
         "added",
@@ -16246,6 +16442,7 @@
         "almost",
         "alone",
         "alongside",
+        "already",
         "ambition",
         "analysis",
         "and",
@@ -16256,6 +16453,8 @@
         "anything",
         "approval",
         "are",
+        "arrived",
+        "attach",
         "attack",
         "bad",
         "baseline",
@@ -16265,10 +16464,12 @@
         "before",
         "build",
         "buys",
+        "calling",
         "can",
         "canary",
         "cannot",
         "cause",
+        "caused",
         "change",
         "changed",
         "changes",
@@ -16279,13 +16480,17 @@
         "commits",
         "compare",
         "compared",
+        "compares",
+        "comparing",
         "computed",
         "confirm",
         "correlate",
         "correlation",
         "count",
         "counting",
+        "counts",
         "dashboard",
+        "dated",
         "decision",
         "definition",
         "definitional",
@@ -16293,11 +16498,12 @@
         "degrades",
         "deliberate",
         "delivery",
-        "denominator",
         "depending",
         "deploy",
         "deployment",
+        "deployments",
         "deploys",
+        "describes",
         "detect",
         "direction",
         "distrust",
@@ -16309,26 +16515,30 @@
         "early",
         "easy",
         "either",
+        "entry",
         "every",
         "everyone",
+        "exist",
         "explain",
+        "fail",
+        "failed",
         "failing",
         "failure",
         "failures",
         "falling",
         "falls",
-        "fast",
+        "feature",
         "fifth",
+        "file",
+        "files",
         "finding",
         "first",
         "five",
         "fix",
-        "fixed",
         "flat",
         "flatters",
         "flowed",
         "follow",
-        "for",
         "forward",
         "frequency",
         "front",
@@ -16340,8 +16550,11 @@
         "gets",
         "give",
         "goes",
+        "grouped",
         "guess",
         "half",
+        "happening",
+        "has",
         "have",
         "health",
         "here",
@@ -16350,18 +16563,20 @@
         "hotfix",
         "hours",
         "how",
+        "inaccurate",
         "incident",
         "incidents",
         "increase",
         "industry",
         "inflate",
+        "instability",
         "instrumentation",
         "into",
         "invite",
         "its",
         "itself",
         "judging",
-        "just",
+        "labelled",
         "lags",
         "last",
         "lead",
@@ -16378,31 +16593,32 @@
         "mature",
         "mean",
         "measure",
-        "measured",
-        "meeting",
-        "meets",
-        "met",
         "meta",
         "metric",
         "metrics",
         "might",
         "minutes",
+        "model",
         "more",
         "move",
         "moved",
         "moves",
         "moving",
-        "negotiated",
+        "names",
+        "narrowing",
         "never",
+        "next",
         "not",
         "number",
         "numbers",
         "one",
         "ones",
         "only",
+        "operational",
         "organization",
         "other",
         "our",
+        "outage",
         "over",
         "own",
         "page",
@@ -16413,13 +16629,16 @@
         "pause",
         "people",
         "perfection",
+        "performance",
         "pick",
         "pipeline",
+        "planned",
         "playbook",
         "practice",
         "prescribing",
         "problem",
         "provenance",
+        "published",
         "quarter",
         "quartiles",
         "quietly",
@@ -16429,12 +16648,16 @@
         "rather",
         "read",
         "reading",
+        "readings",
         "recent",
         "records",
+        "recovery",
         "redefine",
-        "reflecting",
         "regressions",
         "reliability",
+        "rename",
+        "renamed",
+        "report",
         "reported",
         "requires",
         "resolved",
@@ -16442,6 +16665,7 @@
         "restore",
         "restoring",
         "review",
+        "rework",
         "rising",
         "risk",
         "rollbacks",
@@ -16451,24 +16675,25 @@
         "say",
         "see",
         "sensor",
+        "sensors",
         "service",
+        "set",
         "ship",
+        "shipped",
         "shipping",
         "signal",
+        "since",
         "size",
         "small",
         "smaller",
+        "something",
         "split",
         "splitting",
-        "stability",
         "stage",
         "stale",
-        "started",
         "stops",
         "sudden",
-        "system",
         "table",
-        "target",
         "targets",
         "team",
         "teams",
@@ -16481,8 +16706,10 @@
         "them",
         "they",
         "this",
+        "three",
         "through",
-        "tighten",
+        "throughput",
+        "ticket",
         "time",
         "timeline",
         "together",
@@ -16494,9 +16721,11 @@
         "trivial",
         "trouble",
         "twenty",
+        "two",
         "type",
         "unaffected",
         "unannounced",
+        "under",
         "underlying",
         "understanding",
         "unmeasured",
@@ -16505,6 +16734,7 @@
         "uses",
         "usually",
         "verify",
+        "versioned",
         "waits",
         "was",
         "way",
@@ -16521,8 +16751,8 @@
         "window",
         "with",
         "without",
+        "work",
         "worse",
-        "yes",
         "you"
       ],
       "url_path": "sensors/dora-metrics/"
@@ -16898,7 +17128,9 @@
       "see_also_ids": [
         "SO-009b",
         "SO-009c",
-        "SO-009d"
+        "SO-009d",
+        "SO-007e",
+        "SO-015b"
       ],
       "see_also_pages": [],
       "slug": "revert-rate",
@@ -17242,7 +17474,8 @@
       "see_also_ids": [
         "SO-009",
         "SO-009b",
-        "SO-009c"
+        "SO-009c",
+        "SO-015b"
       ],
       "see_also_pages": [],
       "slug": "time-to-repair",
@@ -18358,7 +18591,8 @@
       "see_also_ids": [
         "SO-010b",
         "SO-010c",
-        "SO-010d"
+        "SO-010d",
+        "SO-011d"
       ],
       "see_also_pages": [],
       "slug": "independent-review",