differential-coverage 0.1.0__tar.gz

differential_coverage-0.1.0/PKG-INFO

@@ -0,0 +1,113 @@
+Metadata-Version: 2.4
+Name: differential_coverage
+Version: 0.1.0
+Summary: Compute differential coverage for fuzzer runs.
+Author-email: Valentin Huber <contact@valentinhuber.me>
+License-Expression: MIT
+Project-URL: Repository, https://github.com/valentinhuber/differential_coverage
+Project-URL: Documentation, https://github.com/valentinhuber/differential_coverage#readme
+Keywords: fuzzing,coverage,differential-coverage
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+Requires-Dist: mypy>=1.19.1; extra == "dev"
+Requires-Dist: pre-commit>=4.3.0; extra == "dev"
+Requires-Dist: pytest>=9.0.2; extra == "dev"
+Requires-Dist: ruff>=0.15.0; extra == "dev"
+
+# Differential Coverage
+
+Absolute coverage numbers aren't painting the whole picture: Different fuzzers may reach the same total coverage, but cover different parts of the program. Fuzzer (A) may reach more coverage than fuzzer (B), but (B) may reach coverage that (A) doesn't reach — so it's still valuable.
+
+## Definition
+
+Differential coverage is a measure for this. It was proposed by Leonelli et al. in their paper on TwinFuzz[^1]. This implementation relies on the formulas from the SBFT’25 Competition Report[^2]. There, Crump et al. define a fuzzers overall score compared to others as
+
+$$
+relscore(f,b,s,e) = \left|f_0\in F|e\notin \text{cov}(b,t,s)\forall t\in \text{trials}(f_0,b)\right|
+\times\frac
+{\left|\{t\in \text{trials}(f,b)\ |\ e \in \text{cov}(b,t,s)\}\right|}
+{\left|\{t\in \text{trials}(f,b)\ |\ \text{cov}(b,t,s)\neq \emptyset\}\right|}
+$$
+
+The score of a fuzzer is then
+
+$$
+\text{score}(f,b,s)=\sum_{e\in E}\text{relscore}(f,b,s,e)
+$$
+
+## Explanation
+This can be simplified to the following:
+
+$$
+\text{differential coverage}(f,e) = \text{number of fuzzers that never hit }e
+\times\frac
+{\text{number of trials of }f\text{ that hit }e}
+{\text{number of trials of }f\text{ with non-empty cov}}
+$$
+$$
+\text{score}(f) = \text{sum of differential coverage}(f,e)\text{ over all edges e}
+$$
+
+## Usage
+Assume `<input_dir>` is a directory of directories for each fuzzer, where each fuzzer subdirectory contains coverage files for each trial. Currently, the output format of `afl-showmap` is supported (lines with `edge_id:count`, where we only care if `count > 0`).
+
+PRs for other data formats welcome :)
+
+### Command Line Interface
+```
+differential_coverage <input_dir>
+```
+
+#### Example
+[`tests/sample_coverage`](./tests/sample_coverage/) provides sample coverage data to explain differential coverage. It contains 3 fuzzers and 3 edges:
+- Edge 1 is always hit by all fuzzers
+- Edge 2 is always hit by `fuzzer_c`, sometimes hit by `fuzzer_a`, and never hit by `fuzzer_b`
+- Edge 3 is always hit by `fuzzer_b` and `fuzzer_c`, but only sometimes by `fuzzer_a`
+
+```
+differential_coverage tests/sample_coverage
+```
+then produces
+```
+fuzzer_c: 1.00
+fuzzer_a: 0.50
+fuzzer_b: 0.00
+```
+### API
+
+```python
+from pathlib import Path
+from differential_coverage import (
+    calculate_scores_for_campaign,
+    calculate_differential_coverage_scores,
+)
+
+# From a campaign directory (one subdir per fuzzer, each with coverage files):
+scores = read_campaign_and_calculate_score(Path("path/to/campaign_dir"))
+# -> dict[str, float]: fuzzer name -> score
+
+# From in-memory campaign data (fuzzer -> trial_id -> edge_id -> count):
+campaign = {...}  # dict[str, dict[str, dict[int, int]]]
+scores = calculate_differential_coverage_scores(campaign)
+# -> dict[str, float]: fuzzer name -> score
+```
+
+## Installation
+```bash
+pip install .
+```
+
+## Development
+Install dev dependencies and set up the pre-commit hook (runs a couple of checks before committing):
+```bash
+pip install -e ".[dev]"
+pre-commit install
+```
+
+[^1]: TWINFUZZ: Differential Testing of Video Hardware Acceleration Stacks, https://www.ndss-symposium.org/ndss-paper/twinfuzz-differential-testing-of-video-hardware-acceleration-stacks/
+
+[^2]: SBFT’25 Competition Report — Fuzzing Track, https://ieeexplore.ieee.org/document/11086561

differential_coverage-0.1.0/README.md

@@ -0,0 +1,93 @@
+# Differential Coverage
+
+Absolute coverage numbers aren't painting the whole picture: Different fuzzers may reach the same total coverage, but cover different parts of the program. Fuzzer (A) may reach more coverage than fuzzer (B), but (B) may reach coverage that (A) doesn't reach — so it's still valuable.
+
+## Definition
+
+Differential coverage is a measure for this. It was proposed by Leonelli et al. in their paper on TwinFuzz[^1]. This implementation relies on the formulas from the SBFT’25 Competition Report[^2]. There, Crump et al. define a fuzzers overall score compared to others as
+
+$$
+relscore(f,b,s,e) = \left|f_0\in F|e\notin \text{cov}(b,t,s)\forall t\in \text{trials}(f_0,b)\right|
+\times\frac
+{\left|\{t\in \text{trials}(f,b)\ |\ e \in \text{cov}(b,t,s)\}\right|}
+{\left|\{t\in \text{trials}(f,b)\ |\ \text{cov}(b,t,s)\neq \emptyset\}\right|}
+$$
+
+The score of a fuzzer is then
+
+$$
+\text{score}(f,b,s)=\sum_{e\in E}\text{relscore}(f,b,s,e)
+$$
+
+## Explanation
+This can be simplified to the following:
+
+$$
+\text{differential coverage}(f,e) = \text{number of fuzzers that never hit }e
+\times\frac
+{\text{number of trials of }f\text{ that hit }e}
+{\text{number of trials of }f\text{ with non-empty cov}}
+$$
+$$
+\text{score}(f) = \text{sum of differential coverage}(f,e)\text{ over all edges e}
+$$
+
+## Usage
+Assume `<input_dir>` is a directory of directories for each fuzzer, where each fuzzer subdirectory contains coverage files for each trial. Currently, the output format of `afl-showmap` is supported (lines with `edge_id:count`, where we only care if `count > 0`).
+
+PRs for other data formats welcome :)
+
+### Command Line Interface
+```
+differential_coverage <input_dir>
+```
+
+#### Example
+[`tests/sample_coverage`](./tests/sample_coverage/) provides sample coverage data to explain differential coverage. It contains 3 fuzzers and 3 edges:
+- Edge 1 is always hit by all fuzzers
+- Edge 2 is always hit by `fuzzer_c`, sometimes hit by `fuzzer_a`, and never hit by `fuzzer_b`
+- Edge 3 is always hit by `fuzzer_b` and `fuzzer_c`, but only sometimes by `fuzzer_a`
+
+```
+differential_coverage tests/sample_coverage
+```
+then produces
+```
+fuzzer_c: 1.00
+fuzzer_a: 0.50
+fuzzer_b: 0.00
+```
+### API
+
+```python
+from pathlib import Path
+from differential_coverage import (
+    calculate_scores_for_campaign,
+    calculate_differential_coverage_scores,
+)
+
+# From a campaign directory (one subdir per fuzzer, each with coverage files):
+scores = read_campaign_and_calculate_score(Path("path/to/campaign_dir"))
+# -> dict[str, float]: fuzzer name -> score
+
+# From in-memory campaign data (fuzzer -> trial_id -> edge_id -> count):
+campaign = {...}  # dict[str, dict[str, dict[int, int]]]
+scores = calculate_differential_coverage_scores(campaign)
+# -> dict[str, float]: fuzzer name -> score
+```
+
+## Installation
+```bash
+pip install .
+```
+
+## Development
+Install dev dependencies and set up the pre-commit hook (runs a couple of checks before committing):
+```bash
+pip install -e ".[dev]"
+pre-commit install
+```
+
+[^1]: TWINFUZZ: Differential Testing of Video Hardware Acceleration Stacks, https://www.ndss-symposium.org/ndss-paper/twinfuzz-differential-testing-of-video-hardware-acceleration-stacks/
+
+[^2]: SBFT’25 Competition Report — Fuzzing Track, https://ieeexplore.ieee.org/document/11086561

differential_coverage-0.1.0/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["setuptools>=80.10.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "differential_coverage"
+version = "0.1.0"
+description = "Compute differential coverage for fuzzer runs."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+authors = [{ name = "Valentin Huber", email = "contact@valentinhuber.me" }]
+keywords = ["fuzzing", "coverage", "differential-coverage"]
+classifiers = [
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Topic :: Software Development :: Testing",
+]
+dependencies = []
+
+[project.optional-dependencies]
+dev = ["mypy>=1.19.1", "pre-commit>=4.3.0", "pytest>=9.0.2", "ruff>=0.15.0"]
+
+[project.urls]
+Repository = "https://github.com/valentinhuber/differential_coverage"
+Documentation = "https://github.com/valentinhuber/differential_coverage#readme"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+
+[project.scripts]
+differential_coverage = "differential_coverage:main"

differential_coverage-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

differential_coverage-0.1.0/src/differential_coverage/__init__.py

@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+"""
+Compute relscore (as defined in SBFT'25) from coverage in afl-showmap format (id:count per line).
+Input: directory of subdirectories (one per fuzzer), each containing coverage files.
+"""
+
+import argparse
+from pathlib import Path
+
+
+def read_afl_showmap_file(file: Path) -> dict[int, int]:
+    """Parse one afl-showmap file; return dict of edge ids to counts."""
+    edges = {}
+    for i, line in enumerate(file.read_text().strip().splitlines()):
+        line = line.strip()
+        if not line:
+            continue
+        split = line.split(":")
+        if len(split) != 2:
+            raise ValueError(f"Invalid line {file}:{i}: {line}")
+        id, count = split
+        edges[int(id)] = int(count)
+    return edges
+
+
+def read_fuzzer_dir(path: Path) -> dict[str, dict[int, int]]:
+    """Read all afl-showmap files in a directory; return dict of trial id to dict of edge ids to counts."""
+    trials = {}
+    for file in path.iterdir():
+        if file.is_file():
+            trials[file.name] = read_afl_showmap_file(file)
+        else:
+            raise ValueError(f"Invalid file: {file}")
+    return trials
+
+
+def read_campaign_dir(path: Path) -> dict[str, dict[str, dict[int, int]]]:
+    """Read all fuzzer directories in a campaign directory; return dict of fuzzer name to dict of trial id to dict of edge ids to counts."""
+    campaigns = {}
+    for fuzzer_dir in path.iterdir():
+        if fuzzer_dir.is_dir():
+            campaigns[fuzzer_dir.name] = read_fuzzer_dir(fuzzer_dir)
+        else:
+            raise ValueError(f"Invalid file: {fuzzer_dir}")
+    return campaigns
+
+
+def calculate_fuzzer_score(
+    trials: dict[str, dict[int, int]],
+    all_edges: set[int],
+    fuzzers_that_never_hit_edge: dict[int, set[str]],
+) -> float:
+    score = 0.0
+    for e in all_edges:
+        fuzzers_that_never_hit_e = len(fuzzers_that_never_hit_edge[e])
+        trials_that_hit_e = len([trial for trial in trials.values() if trial.get(e, 0)])
+        trials_with_non_empty_cov = len(
+            [trial for trial in trials.values() if any(trial.values())]
+        )
+        score += (
+            fuzzers_that_never_hit_e * trials_that_hit_e / trials_with_non_empty_cov
+        )
+    return score
+
+
+def calculate_differential_coverage_scores(
+    campaign: dict[str, dict[str, dict[int, int]]],
+) -> dict[str, float]:
+    """
+    differential_coverage(f,e) = (# fuzzers that never hit e) * (# trials of f that hit e) / (# trials of f with non-empty cov).
+    score(f) = sum of differential_coverage(f,e) over all edges e.
+    Returns scores for each fuzzer run.
+    """
+    all_edges = set(
+        edge
+        for fuzzer in campaign.values()
+        for trial in fuzzer.values()
+        for edge in trial.keys()
+    )
+
+    fuzzers_that_never_hit_edge: dict[int, set[str]] = {
+        edge: set(
+            fuzzer
+            for fuzzer, trials in campaign.items()
+            if not any(trial.get(edge, 0) for trial in trials.values())
+        )
+        for edge in all_edges
+    }
+
+    scores = {
+        fuzzer: calculate_fuzzer_score(trials, all_edges, fuzzers_that_never_hit_edge)
+        for fuzzer, trials in campaign.items()
+    }
+    return scores
+
+
+def read_campaign_and_calculate_score(root: Path) -> dict[str, float]:
+    if not root.is_dir():
+        raise ValueError(f"Not a directory: {root}")
+
+    campaign_data = read_campaign_dir(root)
+    return calculate_differential_coverage_scores(campaign_data)
+
+
+def main() -> None:
+    p = argparse.ArgumentParser(
+        description="Compute differential coverage from afl-showmap coverage dirs."
+    )
+    p.add_argument(
+        "dir",
+        type=Path,
+        help="Directory containing one subdir per fuzzer with coverage files",
+    )
+    args = p.parse_args()
+
+    root = args.dir.resolve()
+    scores = read_campaign_and_calculate_score(root)
+    for fuzzer, score in sorted(scores.items(), key=lambda x: x[1], reverse=True):
+        print(f"{fuzzer}: {score:.2f}")
+
+
+if __name__ == "__main__":
+    main()

differential_coverage-0.1.0/src/differential_coverage.egg-info/PKG-INFO

@@ -0,0 +1,113 @@
+Metadata-Version: 2.4
+Name: differential_coverage
+Version: 0.1.0
+Summary: Compute differential coverage for fuzzer runs.
+Author-email: Valentin Huber <contact@valentinhuber.me>
+License-Expression: MIT
+Project-URL: Repository, https://github.com/valentinhuber/differential_coverage
+Project-URL: Documentation, https://github.com/valentinhuber/differential_coverage#readme
+Keywords: fuzzing,coverage,differential-coverage
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+Requires-Dist: mypy>=1.19.1; extra == "dev"
+Requires-Dist: pre-commit>=4.3.0; extra == "dev"
+Requires-Dist: pytest>=9.0.2; extra == "dev"
+Requires-Dist: ruff>=0.15.0; extra == "dev"
+
+# Differential Coverage
+
+Absolute coverage numbers aren't painting the whole picture: Different fuzzers may reach the same total coverage, but cover different parts of the program. Fuzzer (A) may reach more coverage than fuzzer (B), but (B) may reach coverage that (A) doesn't reach — so it's still valuable.
+
+## Definition
+
+Differential coverage is a measure for this. It was proposed by Leonelli et al. in their paper on TwinFuzz[^1]. This implementation relies on the formulas from the SBFT’25 Competition Report[^2]. There, Crump et al. define a fuzzers overall score compared to others as
+
+$$
+relscore(f,b,s,e) = \left|f_0\in F|e\notin \text{cov}(b,t,s)\forall t\in \text{trials}(f_0,b)\right|
+\times\frac
+{\left|\{t\in \text{trials}(f,b)\ |\ e \in \text{cov}(b,t,s)\}\right|}
+{\left|\{t\in \text{trials}(f,b)\ |\ \text{cov}(b,t,s)\neq \emptyset\}\right|}
+$$
+
+The score of a fuzzer is then
+
+$$
+\text{score}(f,b,s)=\sum_{e\in E}\text{relscore}(f,b,s,e)
+$$
+
+## Explanation
+This can be simplified to the following:
+
+$$
+\text{differential coverage}(f,e) = \text{number of fuzzers that never hit }e
+\times\frac
+{\text{number of trials of }f\text{ that hit }e}
+{\text{number of trials of }f\text{ with non-empty cov}}
+$$
+$$
+\text{score}(f) = \text{sum of differential coverage}(f,e)\text{ over all edges e}
+$$
+
+## Usage
+Assume `<input_dir>` is a directory of directories for each fuzzer, where each fuzzer subdirectory contains coverage files for each trial. Currently, the output format of `afl-showmap` is supported (lines with `edge_id:count`, where we only care if `count > 0`).
+
+PRs for other data formats welcome :)
+
+### Command Line Interface
+```
+differential_coverage <input_dir>
+```
+
+#### Example
+[`tests/sample_coverage`](./tests/sample_coverage/) provides sample coverage data to explain differential coverage. It contains 3 fuzzers and 3 edges:
+- Edge 1 is always hit by all fuzzers
+- Edge 2 is always hit by `fuzzer_c`, sometimes hit by `fuzzer_a`, and never hit by `fuzzer_b`
+- Edge 3 is always hit by `fuzzer_b` and `fuzzer_c`, but only sometimes by `fuzzer_a`
+
+```
+differential_coverage tests/sample_coverage
+```
+then produces
+```
+fuzzer_c: 1.00
+fuzzer_a: 0.50
+fuzzer_b: 0.00
+```
+### API
+
+```python
+from pathlib import Path
+from differential_coverage import (
+    calculate_scores_for_campaign,
+    calculate_differential_coverage_scores,
+)
+
+# From a campaign directory (one subdir per fuzzer, each with coverage files):
+scores = read_campaign_and_calculate_score(Path("path/to/campaign_dir"))
+# -> dict[str, float]: fuzzer name -> score
+
+# From in-memory campaign data (fuzzer -> trial_id -> edge_id -> count):
+campaign = {...}  # dict[str, dict[str, dict[int, int]]]
+scores = calculate_differential_coverage_scores(campaign)
+# -> dict[str, float]: fuzzer name -> score
+```
+
+## Installation
+```bash
+pip install .
+```
+
+## Development
+Install dev dependencies and set up the pre-commit hook (runs a couple of checks before committing):
+```bash
+pip install -e ".[dev]"
+pre-commit install
+```
+
+[^1]: TWINFUZZ: Differential Testing of Video Hardware Acceleration Stacks, https://www.ndss-symposium.org/ndss-paper/twinfuzz-differential-testing-of-video-hardware-acceleration-stacks/
+
+[^2]: SBFT’25 Competition Report — Fuzzing Track, https://ieeexplore.ieee.org/document/11086561

differential_coverage-0.1.0/src/differential_coverage.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+README.md
+pyproject.toml
+src/differential_coverage/__init__.py
+src/differential_coverage.egg-info/PKG-INFO
+src/differential_coverage.egg-info/SOURCES.txt
+src/differential_coverage.egg-info/dependency_links.txt
+src/differential_coverage.egg-info/entry_points.txt
+src/differential_coverage.egg-info/requires.txt
+src/differential_coverage.egg-info/top_level.txt
+tests/test_sample_coverage.py

differential_coverage-0.1.0/src/differential_coverage.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

differential_coverage-0.1.0/src/differential_coverage.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+differential_coverage = differential_coverage:main

differential_coverage-0.1.0/src/differential_coverage.egg-info/requires.txt

@@ -0,0 +1,6 @@
+
+[dev]
+mypy>=1.19.1
+pre-commit>=4.3.0
+pytest>=9.0.2
+ruff>=0.15.0

differential_coverage-0.1.0/src/differential_coverage.egg-info/top_level.txt

@@ -0,0 +1 @@
+differential_coverage

differential_coverage-0.1.0/tests/test_sample_coverage.py

@@ -0,0 +1,13 @@
+from pathlib import Path
+from differential_coverage import read_campaign_and_calculate_score
+
+
+def test_sample_coverage() -> None:
+    scores = read_campaign_and_calculate_score(
+        (Path(__file__).parent / "sample_coverage").resolve()
+    )
+    assert scores == {
+        "fuzzer_c": 1.0,
+        "fuzzer_a": 0.5,
+        "fuzzer_b": 0.0,
+    }