whyfull 0.1.0__tar.gz

whyfull-0.1.0/.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+*.pyc
+.pytest_cache/
+.ruff_cache/
+build/
+dist/
+*.egg-info/
+.venv/

whyfull-0.1.0/CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog
+
+## 0.1.0
+
+First release.
+
+- Checks: deleted-but-open files, Docker (build cache, images, containers, volumes, container logs), systemd journal, rotated and oversized logs, old kernels, package manager caches, snap revisions, core dumps, old temp files, user build and package caches, inode exhaustion, and the ext root reserve.
+- Full-filesystem scan with a time limit, largest directories, crowded directories, and a warning when `df` and the files on disk don't add up.
+- Text, JSON and Markdown output; `--fail-above` for cron and monitoring.
+- Runs on Python 3.6+ with no dependencies; also shipped as a single `whyfull.pyz` file.

whyfull-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ahmed Laban
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

whyfull-0.1.0/PKG-INFO

@@ -0,0 +1,148 @@
+Metadata-Version: 2.5
+Name: whyfull
+Version: 0.1.0
+Summary: Find out why your Linux disk is full: deleted-but-open files, Docker, journald, old kernels, caches and more, each with the command that fixes it.
+Project-URL: Homepage, https://github.com/amlaban65/whyfull
+Project-URL: Issues, https://github.com/amlaban65/whyfull/issues
+Project-URL: Changelog, https://github.com/amlaban65/whyfull/blob/main/CHANGELOG.md
+Author: Ahmed Laban
+License: MIT
+License-File: LICENSE
+Keywords: cleanup,deleted files,devops,df,disk full,disk space,docker,du,inodes,journald,linux,lsof,no space left on device,sre,sysadmin
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: System :: Filesystems
+Classifier: Topic :: System :: Monitoring
+Classifier: Topic :: System :: Systems Administration
+Classifier: Topic :: Utilities
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+
+# whyfull
+
+**Find out why your Linux disk is full, and get the command that fixes it.**
+
+[![CI](https://github.com/amlaban65/whyfull/actions/workflows/ci.yml/badge.svg)](https://github.com/amlaban65/whyfull/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/whyfull)](https://pypi.org/project/whyfull/)
+[![Python 3.6+](https://img.shields.io/badge/python-3.6%2B-blue)](https://pypi.org/project/whyfull/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/amlaban65/whyfull/blob/main/LICENSE)
+
+`df` says 100%. `du` doesn't add up. Something is failing with `No space left on device`.
+
+whyfull checks the places where disk space usually goes on a Linux server: deleted files that a process still holds open, the systemd journal, Docker images and container logs, old kernels, package caches, core dumps, runaway logs and inode exhaustion. For each thing it finds, it prints how much space it takes, how risky it is to remove, and the exact command to get the space back.
+
+It never deletes anything itself.
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/amlaban65/whyfull/main/docs/demo.svg" alt="whyfull showing a full root filesystem: a 700 MB deleted nginx log still held open, a 560 MB Docker container log, unused images, old kernels and the systemd journal, each with the command that frees it" width="100%">
+</p>
+
+## Run it
+
+A full disk may leave no room to install anything, so whyfull is a single 30 KB file with no dependencies. It runs on the Python 3 that already ships with nearly every Linux server (3.6 or newer). Download it to `/dev/shm`, which lives in memory:
+
+```sh
+curl -fsSL -o /dev/shm/whyfull https://github.com/amlaban65/whyfull/releases/latest/download/whyfull.pyz
+sudo python3 /dev/shm/whyfull
+```
+
+Or install it from PyPI:
+
+```sh
+pipx install whyfull      # or: uv tool install whyfull, pip install --user whyfull
+sudo "$(command -v whyfull)"
+```
+
+`uvx whyfull` and `pipx run whyfull` work for a one-off run.
+
+Run it with `sudo`. Without root it can't measure files owned by other users and services, and it will tell you what it couldn't read.
+
+## What it looks for
+
+| Check | What it finds |
+|---|---|
+| `deleted-open` | Files that were deleted while a process still had them open. This is the classic "df says full but du says otherwise". whyfull names the systemd service or Docker container holding them and shows how to release the space with or without a restart. |
+| `docker` | Build cache, unused images, stopped containers, dangling volumes, and container logs with no size limit. |
+| `journald` | A systemd journal that has grown past a sensible size, with the commands to vacuum it and cap it. |
+| `logs` | Rotated and compressed logs in `/var/log`, and single log files over 1 GB. |
+| `kernels` | Installed kernels other than the running one and the newest one, with the right removal command for apt, dnf or zypper. |
+| `packages` | apt, dnf, yum, pacman, zypper and apk caches. |
+| `snaps` | Disabled snap revisions that snapd keeps around. |
+| `crashes` | Core dumps from systemd-coredump, apport and ABRT, and `/var/crash`. |
+| `tmp` | Files in `/tmp` and `/var/tmp` nobody has touched in a week. |
+| `user-caches` | pip, uv, poetry, npm, yarn, pnpm, Go, Cargo, Maven, Gradle, Hugging Face, PyTorch and conda caches, and the Trash, for every user. |
+| `inodes` | A filesystem that ran out of inodes (too many files) rather than bytes, and the directories with the most entries. |
+| `reserved` | ext filesystems that look full only because of the 5% reserved for root. |
+
+whyfull also scans the whole filesystem (with a time limit) and lists the largest directories. When the files it can see add up to much less than `df` reports, it says so, and points to where the rest usually is. Common causes are files hidden underneath a mount point and btrfs or ZFS snapshots.
+
+`whyfull --list-checks` prints the list from your installed version.
+
+### Risk levels
+
+Every finding is labelled so you know what you're agreeing to:
+
+- **safe**: caches that are rebuilt on demand. Removing them costs a slower next build or download.
+- **review**: almost always fine to remove (old kernels, rotated logs, unused images), but take a look first.
+- **caution**: can lose data or affect a running service, such as truncating a live log or pruning Docker volumes.
+- **info**: explains what's going on, like inode exhaustion or the root reserve, without a direct cleanup.
+
+## Usage
+
+```sh
+sudo whyfull                      # every filesystem that is 85% full or more
+sudo whyfull /var                 # only the filesystem holding /var
+sudo whyfull -t 70                # lower the "full" threshold to 70%
+sudo whyfull -v                   # explain every finding, list all evidence
+sudo whyfull --no-scan            # skip the full scan, run only the checks
+sudo whyfull --only docker,logs   # run some checks; see --list-checks
+```
+
+If nothing is above the threshold, whyfull analyses `/` anyway and tells you the disks are fine.
+
+### For incident tickets, scripts and monitoring
+
+```sh
+sudo whyfull --format markdown > disk.md   # paste into a ticket or chat
+sudo whyfull --json | jq '.targets[].findings[] | {title, bytes, risk}'
+whyfull --fail-above 90 --no-scan          # exits 1 when any filesystem is 90%+ full
+```
+
+Exit codes: `0` when the run finished, `1` when `--fail-above` was exceeded, `2` for usage errors.
+
+## How is this different from ncdu, du, dust or gdu?
+
+Those are great for browsing a directory tree, and you should keep using them. They answer "which folder is big". whyfull answers "what is this space, and how do I get it back without breaking anything". It finds space that no directory browser can see (deleted files held open by a process). It knows that `/var/lib/docker/overlay2` is cleaned with `docker image prune`, not `rm`. It knows which kernels are safe to remove, and that a 2 GB `syslog.2.gz` can go while a live `syslog` should be truncated.
+
+Run whyfull first. If something is still unexplained, open ncdu on the directory it points to.
+
+## Safety
+
+- whyfull only reads. It does not delete, truncate, prune or restart anything.
+- The only commands it runs are read-only: `docker system df`, `docker info` and `snap list --all`.
+- Every fix is printed for you to read and run yourself. Commands that can lose data are marked **caution** and explained.
+- Scans stay on one filesystem (like `du -x`), count hard links once, and stop at a time limit, so they won't hang on a huge disk or wander into network mounts.
+
+## Contributing
+
+The most useful contribution is a new check: *"my disk filled up because of X, and here is how to spot it and clean it up."* Each check is one short file in [`src/whyfull/checks/`](https://github.com/amlaban65/whyfull/tree/main/src/whyfull/checks), and tests run against a fake filesystem, so you don't need a full disk to write one. [CONTRIBUTING.md](https://github.com/amlaban65/whyfull/blob/main/CONTRIBUTING.md) walks through it.
+
+Some checks that would help a lot of people: containerd and CRI-O images (`crictl`), Podman, Kubernetes pod logs, PostgreSQL WAL, MySQL binlogs, Elasticsearch, LXD, Flatpak, Jenkins workspaces and GitLab Runner caches. If you have a war story about a disk that filled up in a strange way, [open an issue](https://github.com/amlaban65/whyfull/issues/new?template=new-check.yml) even if you don't plan to write the code.
+
+## License
+
+[MIT](https://github.com/amlaban65/whyfull/blob/main/LICENSE)

whyfull-0.1.0/README.md

@@ -0,0 +1,114 @@
+# whyfull
+
+**Find out why your Linux disk is full, and get the command that fixes it.**
+
+[![CI](https://github.com/amlaban65/whyfull/actions/workflows/ci.yml/badge.svg)](https://github.com/amlaban65/whyfull/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/whyfull)](https://pypi.org/project/whyfull/)
+[![Python 3.6+](https://img.shields.io/badge/python-3.6%2B-blue)](https://pypi.org/project/whyfull/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/amlaban65/whyfull/blob/main/LICENSE)
+
+`df` says 100%. `du` doesn't add up. Something is failing with `No space left on device`.
+
+whyfull checks the places where disk space usually goes on a Linux server: deleted files that a process still holds open, the systemd journal, Docker images and container logs, old kernels, package caches, core dumps, runaway logs and inode exhaustion. For each thing it finds, it prints how much space it takes, how risky it is to remove, and the exact command to get the space back.
+
+It never deletes anything itself.
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/amlaban65/whyfull/main/docs/demo.svg" alt="whyfull showing a full root filesystem: a 700 MB deleted nginx log still held open, a 560 MB Docker container log, unused images, old kernels and the systemd journal, each with the command that frees it" width="100%">
+</p>
+
+## Run it
+
+A full disk may leave no room to install anything, so whyfull is a single 30 KB file with no dependencies. It runs on the Python 3 that already ships with nearly every Linux server (3.6 or newer). Download it to `/dev/shm`, which lives in memory:
+
+```sh
+curl -fsSL -o /dev/shm/whyfull https://github.com/amlaban65/whyfull/releases/latest/download/whyfull.pyz
+sudo python3 /dev/shm/whyfull
+```
+
+Or install it from PyPI:
+
+```sh
+pipx install whyfull      # or: uv tool install whyfull, pip install --user whyfull
+sudo "$(command -v whyfull)"
+```
+
+`uvx whyfull` and `pipx run whyfull` work for a one-off run.
+
+Run it with `sudo`. Without root it can't measure files owned by other users and services, and it will tell you what it couldn't read.
+
+## What it looks for
+
+| Check | What it finds |
+|---|---|
+| `deleted-open` | Files that were deleted while a process still had them open. This is the classic "df says full but du says otherwise". whyfull names the systemd service or Docker container holding them and shows how to release the space with or without a restart. |
+| `docker` | Build cache, unused images, stopped containers, dangling volumes, and container logs with no size limit. |
+| `journald` | A systemd journal that has grown past a sensible size, with the commands to vacuum it and cap it. |
+| `logs` | Rotated and compressed logs in `/var/log`, and single log files over 1 GB. |
+| `kernels` | Installed kernels other than the running one and the newest one, with the right removal command for apt, dnf or zypper. |
+| `packages` | apt, dnf, yum, pacman, zypper and apk caches. |
+| `snaps` | Disabled snap revisions that snapd keeps around. |
+| `crashes` | Core dumps from systemd-coredump, apport and ABRT, and `/var/crash`. |
+| `tmp` | Files in `/tmp` and `/var/tmp` nobody has touched in a week. |
+| `user-caches` | pip, uv, poetry, npm, yarn, pnpm, Go, Cargo, Maven, Gradle, Hugging Face, PyTorch and conda caches, and the Trash, for every user. |
+| `inodes` | A filesystem that ran out of inodes (too many files) rather than bytes, and the directories with the most entries. |
+| `reserved` | ext filesystems that look full only because of the 5% reserved for root. |
+
+whyfull also scans the whole filesystem (with a time limit) and lists the largest directories. When the files it can see add up to much less than `df` reports, it says so, and points to where the rest usually is. Common causes are files hidden underneath a mount point and btrfs or ZFS snapshots.
+
+`whyfull --list-checks` prints the list from your installed version.
+
+### Risk levels
+
+Every finding is labelled so you know what you're agreeing to:
+
+- **safe**: caches that are rebuilt on demand. Removing them costs a slower next build or download.
+- **review**: almost always fine to remove (old kernels, rotated logs, unused images), but take a look first.
+- **caution**: can lose data or affect a running service, such as truncating a live log or pruning Docker volumes.
+- **info**: explains what's going on, like inode exhaustion or the root reserve, without a direct cleanup.
+
+## Usage
+
+```sh
+sudo whyfull                      # every filesystem that is 85% full or more
+sudo whyfull /var                 # only the filesystem holding /var
+sudo whyfull -t 70                # lower the "full" threshold to 70%
+sudo whyfull -v                   # explain every finding, list all evidence
+sudo whyfull --no-scan            # skip the full scan, run only the checks
+sudo whyfull --only docker,logs   # run some checks; see --list-checks
+```
+
+If nothing is above the threshold, whyfull analyses `/` anyway and tells you the disks are fine.
+
+### For incident tickets, scripts and monitoring
+
+```sh
+sudo whyfull --format markdown > disk.md   # paste into a ticket or chat
+sudo whyfull --json | jq '.targets[].findings[] | {title, bytes, risk}'
+whyfull --fail-above 90 --no-scan          # exits 1 when any filesystem is 90%+ full
+```
+
+Exit codes: `0` when the run finished, `1` when `--fail-above` was exceeded, `2` for usage errors.
+
+## How is this different from ncdu, du, dust or gdu?
+
+Those are great for browsing a directory tree, and you should keep using them. They answer "which folder is big". whyfull answers "what is this space, and how do I get it back without breaking anything". It finds space that no directory browser can see (deleted files held open by a process). It knows that `/var/lib/docker/overlay2` is cleaned with `docker image prune`, not `rm`. It knows which kernels are safe to remove, and that a 2 GB `syslog.2.gz` can go while a live `syslog` should be truncated.
+
+Run whyfull first. If something is still unexplained, open ncdu on the directory it points to.
+
+## Safety
+
+- whyfull only reads. It does not delete, truncate, prune or restart anything.
+- The only commands it runs are read-only: `docker system df`, `docker info` and `snap list --all`.
+- Every fix is printed for you to read and run yourself. Commands that can lose data are marked **caution** and explained.
+- Scans stay on one filesystem (like `du -x`), count hard links once, and stop at a time limit, so they won't hang on a huge disk or wander into network mounts.
+
+## Contributing
+
+The most useful contribution is a new check: *"my disk filled up because of X, and here is how to spot it and clean it up."* Each check is one short file in [`src/whyfull/checks/`](https://github.com/amlaban65/whyfull/tree/main/src/whyfull/checks), and tests run against a fake filesystem, so you don't need a full disk to write one. [CONTRIBUTING.md](https://github.com/amlaban65/whyfull/blob/main/CONTRIBUTING.md) walks through it.
+
+Some checks that would help a lot of people: containerd and CRI-O images (`crictl`), Podman, Kubernetes pod logs, PostgreSQL WAL, MySQL binlogs, Elasticsearch, LXD, Flatpak, Jenkins workspaces and GitLab Runner caches. If you have a war story about a disk that filled up in a strange way, [open an issue](https://github.com/amlaban65/whyfull/issues/new?template=new-check.yml) even if you don't plan to write the code.
+
+## License
+
+[MIT](https://github.com/amlaban65/whyfull/blob/main/LICENSE)

whyfull-0.1.0/pyproject.toml

@@ -0,0 +1,68 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "whyfull"
+dynamic = ["version"]
+description = "Find out why your Linux disk is full: deleted-but-open files, Docker, journald, old kernels, caches and more, each with the command that fixes it."
+readme = "README.md"
+license = { text = "MIT" }
+authors = [{ name = "Ahmed Laban" }]
+requires-python = ">=3.6"
+dependencies = []
+keywords = [
+    "disk full", "disk space", "no space left on device", "df", "du", "linux", "sysadmin",
+    "devops", "sre", "docker", "journald", "inodes", "deleted files", "lsof", "cleanup",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: System Administrators",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.6",
+    "Programming Language :: Python :: 3.7",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: System :: Filesystems",
+    "Topic :: System :: Monitoring",
+    "Topic :: System :: Systems Administration",
+    "Topic :: Utilities",
+]
+
+[project.urls]
+Homepage = "https://github.com/amlaban65/whyfull"
+Issues = "https://github.com/amlaban65/whyfull/issues"
+Changelog = "https://github.com/amlaban65/whyfull/blob/main/CHANGELOG.md"
+
+[project.scripts]
+whyfull = "whyfull.cli:main"
+
+[tool.hatch.version]
+path = "src/whyfull/__init__.py"
+
+[tool.hatch.build.targets.sdist]
+include = ["src", "tests", "tools", "README.md", "LICENSE", "CHANGELOG.md"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/whyfull"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py37"  # ruff's oldest target; vermin in CI checks 3.6
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "B", "UP"]
+# Keep the code runnable on Python 3.6: no f-strings, no newer syntax.
+ignore = ["UP021", "UP022", "UP004", "UP008", "UP009", "UP010", "UP015", "UP025", "UP031", "UP032", "B904"]

whyfull-0.1.0/src/whyfull/__init__.py

@@ -0,0 +1,3 @@
+"""whyfull: find out why a Linux disk is full, and how to safely free space."""
+
+__version__ = "0.1.0"

whyfull-0.1.0/src/whyfull/__main__.py

@@ -0,0 +1,5 @@
+import sys
+
+from whyfull.cli import main
+
+sys.exit(main())

whyfull-0.1.0/src/whyfull/analyze.py

@@ -0,0 +1,167 @@
+"""Run the checks against the chosen filesystems and assemble a report."""
+
+import os
+
+from whyfull import __version__
+from whyfull.checks import Context, load
+from whyfull.model import CAUTION, INFO, REVIEW, SAFE, Finding
+from whyfull.mounts import containing, filesystems
+from whyfull.units import human
+from whyfull.walk import Budget, scan
+
+DEFAULT_THRESHOLD = 85.0
+
+
+class Target(object):
+    """One filesystem under analysis and everything found about it."""
+
+    def __init__(self, fs):
+        self.fs = fs
+        self.findings = []
+        self.tree = None
+        self.largest = []  # [(display path, bytes, [(child path, bytes)])]
+        self.crowded = []  # [(entries, display path)]
+
+    def reclaimable(self):
+        out = {SAFE: 0, REVIEW: 0, CAUTION: 0}
+        for f in self.findings:
+            if f.risk in out:
+                out[f.risk] += f.bytes
+        out["total"] = out[SAFE] + out[REVIEW] + out[CAUTION]
+        return out
+
+    def largest_dirs(self, system, top=6, children=3, min_share=0.01):
+        """[(display path, bytes, [(child display path, bytes)])] from the scan."""
+        if self.tree is None:
+            return []
+        root = system.path(self.fs.mountpoint).rstrip("/") or "/"
+        floor = self.fs.used * min_share
+
+        def kids(parent):
+            out = [(p, b) for p, b in self.tree.sizes.items()
+                   if os.path.dirname(p) == parent and p != parent and b >= floor]
+            return sorted(out, key=lambda pb: -pb[1])
+
+        result = []
+        for path, nbytes in kids(root)[:top]:
+            sub = [(system.display(p), b) for p, b in kids(path)[:children]]
+            result.append((system.display(path), nbytes, sub))
+        return result
+
+
+class Report(object):
+    def __init__(self):
+        self.version = __version__
+        self.filesystems = []
+        self.targets = []
+        self.threshold = DEFAULT_THRESHOLD
+        self.all_clear = False  # no filesystem above the threshold
+        self.notes = []
+        self.errors = []  # (check id, message)
+
+
+def choose_targets(system, all_fs, paths, threshold):
+    if paths:
+        chosen = []
+        for p in paths:
+            fs = containing(system, os.path.abspath(p) if system.root == "/" else p)
+            if fs is None:
+                raise ValueError("can't find the filesystem for %s" % p)
+            if fs.dev not in [t.dev for t in chosen]:
+                chosen.append(fs)
+        return chosen, False
+    full = [fs for fs in all_fs if fs.percent >= threshold]
+    if full:
+        return sorted(full, key=lambda f: -f.percent), False
+    root = containing(system, "/", all_fs) or containing(system, "/")
+    return ([root] if root else []), True
+
+
+def analyze(system, paths=None, threshold=DEFAULT_THRESHOLD, scan_seconds=30, check_seconds=60,
+            only=None, skip=None, do_scan=True):
+    report = Report()
+    report.threshold = threshold
+    all_fs = filesystems(system)
+    report.filesystems = all_fs
+    targets, report.all_clear = choose_targets(system, all_fs, paths, threshold)
+    report.targets = [Target(fs) for fs in targets]
+    if not targets:
+        return report
+
+    known = {fs.dev: fs for fs in all_fs}
+    for fs in targets:
+        known.setdefault(fs.dev, fs)
+    ctx = Context(system, targets, Budget(check_seconds), list(known.values()))
+
+    for chk in load():
+        if only and chk.id not in only:
+            continue
+        if skip and chk.id in skip:
+            continue
+        try:
+            found = list(chk.fn(ctx) or [])
+        except Exception as e:  # a broken check must never hide the others
+            report.errors.append((chk.id, "%s: %s" % (type(e).__name__, e)))
+            continue
+        by_dev = {t.fs.dev: t for t in report.targets}
+        for f in found:
+            t = by_dev.get(f.dev)
+            if t is not None and (f.bytes > 0 or f.risk == INFO):
+                t.findings.append(f)
+    if ctx.budget.hit:
+        ctx.note("Some checks hit their time limit; sizes may be understated.")
+
+    for t in report.targets:
+        if do_scan:
+            t.tree = scan(system.path(t.fs.mountpoint), t.fs.dev, Budget(scan_seconds))
+            if not t.tree.complete:
+                ctx.note("The scan of %s stopped at its time limit; raise it with --scan-time." % t.fs.mountpoint)
+            if t.tree.denied and not system.is_root:
+                ctx.note("%d directories on %s couldn't be read; run with sudo for a complete picture."
+                         % (t.tree.denied, t.fs.mountpoint))
+            t.largest = t.largest_dirs(system)
+            t.crowded = [(n, system.display(p)) for n, p in t.tree.crowded]
+            gap = unexplained(system, t, ctx.deleted_open_bytes.get(t.fs.dev, 0), all_fs)
+            if gap is not None:
+                t.findings.append(gap)
+        t.findings.sort(key=lambda f: (-f.priority, -f.bytes))
+
+    report.notes = ctx.notes
+    return report
+
+
+def unexplained(system, target, deleted_open, all_fs):
+    """An INFO finding when df's 'used' is far above what the files add up to."""
+    tree, fs = target.tree, target.fs
+    if tree is None or not tree.complete or tree.denied:
+        return None
+    gap = fs.used - tree.total - deleted_open
+    if gap < max(512 * 2 ** 20, 0.05 * fs.used):
+        return None
+    root = fs.mountpoint.rstrip("/")
+    covering = sorted(m.mountpoint for m in all_fs
+                      if m.dev != fs.dev and m.mountpoint != fs.mountpoint
+                      and (m.mountpoint + "/").startswith(root + "/"))
+    fix = []
+    if covering:
+        fix = ["sudo mkdir -p /mnt/whyfull-under && sudo mount --bind %s /mnt/whyfull-under" % fs.mountpoint]
+        fix += ["sudo du -sh /mnt/whyfull-under%s" % m[len(root):] for m in covering[:6]]
+        fix += ["sudo umount /mnt/whyfull-under"]
+    if fs.fstype == "btrfs":
+        fix += ["sudo btrfs subvolume list -s %s  # snapshots keep deleted data" % fs.mountpoint]
+    if fs.fstype == "zfs":
+        fix += ["zfs list -t snapshot -o name,used -s used"]
+    causes = []
+    if covering:
+        causes.append("files written into a directory before another disk was mounted over it "
+                      "(%s): they still use space but are hidden" % ", ".join(covering[:4]))
+    if fs.fstype in ("btrfs", "zfs"):
+        causes.append("snapshots holding on to deleted data")
+    causes.append("filesystem metadata and journal")
+    return Finding(
+        "unexplained", fs.mountpoint,
+        "%s in use on %s isn't in any file whyfull can see" % (human(gap), fs.mountpoint),
+        gap, INFO, fs.dev, priority=10,
+        detail="Likely causes: " + "; ".join(causes) + ".",
+        fix=fix,
+    )

whyfull-0.1.0/src/whyfull/checks/__init__.py

@@ -0,0 +1,92 @@
+"""Checks find specific, explainable causes of disk usage.
+
+A check is a function decorated with @check. It receives a Context and
+returns (or yields) Finding objects. Checks must never modify the system:
+they read files and run read-only commands only.
+
+To add one, create a module in this package, decorate a function with
+@check("my-id", "What it looks for"), and add the module name to MODULES
+below. See CONTRIBUTING.md for the full walkthrough.
+"""
+
+import importlib
+
+from whyfull.walk import du
+
+REGISTRY = []
+
+MODULES = [
+    "deleted_open",
+    "space",
+    "journald",
+    "logs",
+    "containers",
+    "packages",
+    "kernels",
+    "snaps",
+    "crashes",
+    "tmpfiles",
+    "usercaches",
+]
+
+
+class Check(object):
+    def __init__(self, id, title, fn):
+        self.id = id
+        self.title = title
+        self.fn = fn
+
+
+def check(id, title):
+    def register(fn):
+        REGISTRY.append(Check(id, title, fn))
+        return fn
+    return register
+
+
+def load():
+    for name in MODULES:
+        importlib.import_module("whyfull.checks." + name)
+    return list(REGISTRY)
+
+
+class Context(object):
+    """What a check gets to work with.
+
+    system   -- whyfull.system.System; use it for every file and command access
+    targets  -- the Filesystem objects being analysed
+    budget   -- shared time budget for size measurements
+    """
+
+    def __init__(self, system, targets, budget, all_filesystems=None):
+        self.system = system
+        self.targets = targets
+        self.budget = budget
+        self.all_filesystems = all_filesystems or targets
+        self.target_devs = set(fs.dev for fs in targets)
+        self.notes = []
+        self.deleted_open_bytes = {}  # dev -> bytes, filled by the deleted_open check
+        self._seen = set()
+
+    def dev_of(self, path):
+        st = self.system.lstat(path)
+        return None if st is None else st.st_dev
+
+    def on_target(self, path):
+        return self.dev_of(path) in self.target_devs
+
+    def size(self, path):
+        """Allocated bytes under path (on its own filesystem), or 0 if missing."""
+        local = self.system.path(path)
+        usage = du(local, budget=self.budget)
+        return usage.bytes
+
+    def filesystem(self, dev):
+        for fs in self.all_filesystems:
+            if fs.dev == dev:
+                return fs
+        return None
+
+    def note(self, text):
+        if text not in self.notes:
+            self.notes.append(text)