similar-files 0.1.0__tar.gz

similar_files-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 craftware
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

similar_files-0.1.0/PKG-INFO

@@ -0,0 +1,96 @@
+Metadata-Version: 2.4
+Name: similar-files
+Version: 0.1.0
+Summary: Find identical and similar files, and write each group as a playlist
+Author: craftware
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/crftwr/similar-files
+Project-URL: Repository, https://github.com/crftwr/similar-files
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Environment :: Console
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Filesystems
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: image
+Requires-Dist: Pillow>=10; extra == "image"
+Requires-Dist: ImageHash>=4.3; extra == "image"
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Dynamic: license-file
+
+# similar-files
+
+Find files that are the same, or that look the same, and get each group as a
+playlist you can open anywhere.
+
+> **Status: early development.** The command line and the library may still
+> change between releases.
+
+## What it finds
+
+- **Identical files**: the same bytes under different names or in different
+  folders. Needs nothing beyond Python.
+- **Similar images**: the same picture resized, re-encoded or lightly edited.
+  Uses an optional dependency.
+- **Similar videos**: the same video re-encoded, resized or re-muxed. Needs
+  [ffmpeg](https://ffmpeg.org) on your `PATH`.
+- **Similar audio**: the same recording re-encoded, or with its tags changed.
+  Needs `fpcalc` from [Chromaprint](https://acoustid.org/chromaprint) on your
+  `PATH` (`brew install chromaprint`, `apt install libchromaprint-tools`).
+
+Content-aware matching ("screenshots of receipts") is planned for later.
+
+## How it works
+
+```bash
+pip install similar-files            # identical files only
+pip install "similar-files[image]"   # plus similar images
+
+similar-files scan ~/Pictures --out ~/similar-groups              # identical files
+similar-files scan ~/Pictures --out ~/similar-images -m image     # similar images
+similar-files scan ~/Movies --out ~/similar-videos -m video       # similar videos
+similar-files scan ~/Music --out ~/similar-music -m audio         # similar audio
+similar-files scan ~/Pictures --out ~/copies -r ~/Desktop/logo.png  # copies of one file
+```
+
+From a checkout, run `python -m similar_files` instead of `similar-files`.
+`--threshold` (0–1) sets how close a match must be, and
+`similar-files extractors` lists what can be compared.
+
+Each group becomes one `.m3u8` playlist in the output folder, and its first
+line is the file the others were compared with:
+
+```
+~/similar-groups/
+  IMG_0012.jpg (3).m3u8
+  logo_final.png (2).m3u8
+```
+
+M3U is a plain, widely supported list format, so you review a group in
+whatever tool you like: a media player,
+[XeFM](https://github.com/crftwr/xefm) (press Enter on a playlist to see its
+files in a pane and copy, move or delete them there), or a text editor.
+
+Two guarantees:
+
+- **similar-files never deletes, moves or changes your files.** It only reads
+  them, and you decide what to keep.
+- **It writes nothing next to your files.** Features it computes are cached in
+  your user cache folder, so a second scan is fast. You can delete that cache
+  at any time.
+
+## Using it from Python
+
+The command line is a thin wrapper. The scanning, caching and grouping are a
+library that other programs can call, and XeFM is the first program to use
+it. See [doc/LIBRARY.md](https://github.com/crftwr/similar-files/blob/main/doc/LIBRARY.md), and for the formats other programs
+read, [doc/PLAYLIST_FORMAT.md](https://github.com/crftwr/similar-files/blob/main/doc/PLAYLIST_FORMAT.md) and
+[doc/CACHE.md](https://github.com/crftwr/similar-files/blob/main/doc/CACHE.md).
+
+## License
+
+MIT. See [LICENSE](https://github.com/crftwr/similar-files/blob/main/LICENSE).

similar_files-0.1.0/README.md

@@ -0,0 +1,72 @@
+# similar-files
+
+Find files that are the same, or that look the same, and get each group as a
+playlist you can open anywhere.
+
+> **Status: early development.** The command line and the library may still
+> change between releases.
+
+## What it finds
+
+- **Identical files**: the same bytes under different names or in different
+  folders. Needs nothing beyond Python.
+- **Similar images**: the same picture resized, re-encoded or lightly edited.
+  Uses an optional dependency.
+- **Similar videos**: the same video re-encoded, resized or re-muxed. Needs
+  [ffmpeg](https://ffmpeg.org) on your `PATH`.
+- **Similar audio**: the same recording re-encoded, or with its tags changed.
+  Needs `fpcalc` from [Chromaprint](https://acoustid.org/chromaprint) on your
+  `PATH` (`brew install chromaprint`, `apt install libchromaprint-tools`).
+
+Content-aware matching ("screenshots of receipts") is planned for later.
+
+## How it works
+
+```bash
+pip install similar-files            # identical files only
+pip install "similar-files[image]"   # plus similar images
+
+similar-files scan ~/Pictures --out ~/similar-groups              # identical files
+similar-files scan ~/Pictures --out ~/similar-images -m image     # similar images
+similar-files scan ~/Movies --out ~/similar-videos -m video       # similar videos
+similar-files scan ~/Music --out ~/similar-music -m audio         # similar audio
+similar-files scan ~/Pictures --out ~/copies -r ~/Desktop/logo.png  # copies of one file
+```
+
+From a checkout, run `python -m similar_files` instead of `similar-files`.
+`--threshold` (0–1) sets how close a match must be, and
+`similar-files extractors` lists what can be compared.
+
+Each group becomes one `.m3u8` playlist in the output folder, and its first
+line is the file the others were compared with:
+
+```
+~/similar-groups/
+  IMG_0012.jpg (3).m3u8
+  logo_final.png (2).m3u8
+```
+
+M3U is a plain, widely supported list format, so you review a group in
+whatever tool you like: a media player,
+[XeFM](https://github.com/crftwr/xefm) (press Enter on a playlist to see its
+files in a pane and copy, move or delete them there), or a text editor.
+
+Two guarantees:
+
+- **similar-files never deletes, moves or changes your files.** It only reads
+  them, and you decide what to keep.
+- **It writes nothing next to your files.** Features it computes are cached in
+  your user cache folder, so a second scan is fast. You can delete that cache
+  at any time.
+
+## Using it from Python
+
+The command line is a thin wrapper. The scanning, caching and grouping are a
+library that other programs can call, and XeFM is the first program to use
+it. See [doc/LIBRARY.md](https://github.com/crftwr/similar-files/blob/main/doc/LIBRARY.md), and for the formats other programs
+read, [doc/PLAYLIST_FORMAT.md](https://github.com/crftwr/similar-files/blob/main/doc/PLAYLIST_FORMAT.md) and
+[doc/CACHE.md](https://github.com/crftwr/similar-files/blob/main/doc/CACHE.md).
+
+## License
+
+MIT. See [LICENSE](https://github.com/crftwr/similar-files/blob/main/LICENSE).

similar_files-0.1.0/pyproject.toml

@@ -0,0 +1,42 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "similar-files"
+description = "Find identical and similar files, and write each group as a playlist"
+readme = "README.md"
+license = "MIT"
+authors = [{ name = "craftware" }]
+requires-python = ">=3.10"
+dynamic = ["version"]
+# Core: standard library only. Layer 1 (identical files) needs nothing more.
+dependencies = []
+classifiers = [
+    "Development Status :: 2 - Pre-Alpha",
+    "Environment :: Console",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Topic :: System :: Filesystems",
+    "Topic :: Utilities",
+]
+
+[project.optional-dependencies]
+image = ["Pillow>=10", "ImageHash>=4.3"]
+test = ["pytest"]
+
+[project.scripts]
+similar-files = "similar_files.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/crftwr/similar-files"
+Repository = "https://github.com/crftwr/similar-files"
+
+[tool.setuptools.packages.find]
+include = ["similar_files*"]
+
+[tool.setuptools.dynamic]
+version = { attr = "similar_files.__version__" }
+
+[tool.pytest.ini_options]
+testpaths = ["test"]

similar_files-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

similar_files-0.1.0/similar_files/__init__.py

@@ -0,0 +1,51 @@
+"""similar-files: find files that are the same, or look the same, and write each group as a playlist.
+
+The public API is what this module exports. Everything else may change.
+"""
+
+from .cache import Cache, CacheError, default_cache_dir, resolve_cache_path
+from .identical import find_identical
+from .model import Group, Member, Progress, ScanResult
+from .playlist import our_playlists, read_playlist, remove_playlists, write_playlists
+from .registry import (
+    ExtractionFailed,
+    Extractor,
+    ExtractorUnavailable,
+    extractors,
+    get_extractor,
+    register,
+)
+from .similar import FeatureSet, extract_features, find_similar, group_features
+from .source import FileItem, LocalFile, WalkStats, walk
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "Cache",
+    "CacheError",
+    "ExtractionFailed",
+    "Extractor",
+    "ExtractorUnavailable",
+    "FeatureSet",
+    "FileItem",
+    "Group",
+    "LocalFile",
+    "Member",
+    "Progress",
+    "ScanResult",
+    "WalkStats",
+    "default_cache_dir",
+    "extract_features",
+    "extractors",
+    "find_identical",
+    "find_similar",
+    "get_extractor",
+    "group_features",
+    "our_playlists",
+    "read_playlist",
+    "register",
+    "remove_playlists",
+    "resolve_cache_path",
+    "walk",
+    "write_playlists",
+]

similar_files-0.1.0/similar_files/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

similar_files-0.1.0/similar_files/_parallel.py

@@ -0,0 +1,112 @@
+"""Thread pools for extraction, with cancellation and progress on the caller's thread."""
+
+from __future__ import annotations
+
+import concurrent.futures as cf
+import os
+import threading
+from typing import Callable, Optional, TypeVar
+
+from .model import CancelCheck, Progress, ProgressCallback
+
+T = TypeVar("T")
+
+#: Workers for local files: capped by the CPU count.
+DEFAULT_MAX_WORKERS = 8
+#: Workers for remote files: a server is not a CPU.
+DEFAULT_REMOTE_WORKERS = 4
+
+_POLL_SECONDS = 0.1
+
+
+def default_workers() -> int:
+    return max(1, min(DEFAULT_MAX_WORKERS, os.cpu_count() or 1))
+
+
+class Runner:
+    """Two thread pools (local and remote items), a cancel flag, and progress counting.
+
+    Jobs run on the pools; waiting, progress callbacks and the cancel check
+    run on the thread that owns the runner.
+    """
+
+    def __init__(
+        self,
+        *,
+        workers: Optional[int] = None,
+        remote_workers: Optional[int] = None,
+        progress: Optional[ProgressCallback] = None,
+        cancel: Optional[CancelCheck] = None,
+    ):
+        self._local = cf.ThreadPoolExecutor(max_workers=workers or default_workers(), thread_name_prefix="sf-local")
+        self._remote_workers = remote_workers or DEFAULT_REMOTE_WORKERS
+        self._remote: Optional[cf.ThreadPoolExecutor] = None
+        self._progress = progress
+        self._cancel = cancel
+        #: Set when the scan is cancelled; workers check it between chunks.
+        self.stop = threading.Event()
+        self._lock = threading.Lock()
+        self._phase = ""
+        self._done = 0
+        self._total = 0
+        self._reported = (None, -1, -1)
+
+    def submit(self, remote: bool, fn: Callable[..., T], *args) -> "cf.Future[T]":
+        if remote:
+            if self._remote is None:
+                self._remote = cf.ThreadPoolExecutor(max_workers=self._remote_workers, thread_name_prefix="sf-remote")
+            pool = self._remote
+        else:
+            pool = self._local
+        future = pool.submit(fn, *args)
+        future.add_done_callback(self._count)
+        return future
+
+    def start_phase(self, phase: str, total: int) -> None:
+        with self._lock:
+            self._phase, self._done, self._total = phase, 0, total
+        self.report()
+
+    def _count(self, _future) -> None:
+        with self._lock:
+            self._done += 1
+
+    def report(self) -> None:
+        if self._progress is None:
+            return
+        with self._lock:
+            state = (self._phase, self._done, self._total)
+        if state != self._reported:
+            self._reported = state
+            self._progress(Progress(*state))
+
+    def cancelled(self) -> bool:
+        """Poll the caller's cancel check; once true, stays true."""
+        if not self.stop.is_set() and self._cancel is not None and self._cancel():
+            self.stop.set()
+        return self.stop.is_set()
+
+    def wait(self, futures) -> bool:
+        """Wait for ``futures``, reporting progress. False if the scan was cancelled first."""
+        pending = set(futures)
+        while pending:
+            if self.cancelled():
+                return False
+            _, pending = cf.wait(pending, timeout=_POLL_SECONDS)
+            self.report()
+        return not self.cancelled()
+
+    def shutdown(self) -> None:
+        """Stop handing out work. Running jobs see :attr:`stop` if cancelled, and are not waited for."""
+        cancelled = self.stop.is_set()
+        for pool in (self._local, self._remote):
+            if pool is not None:
+                pool.shutdown(wait=not cancelled, cancel_futures=True)
+
+    def __enter__(self) -> "Runner":
+        return self
+
+    def __exit__(self, *exc) -> None:
+        if exc[0] is not None:
+            self.stop.set()
+        self.shutdown()

similar_files-0.1.0/similar_files/audio.py

@@ -0,0 +1,115 @@
+"""Layer 2 for audio: Chromaprint fingerprints (``fpcalc``).
+
+Needs ``fpcalc`` on ``PATH`` (Chromaprint: ``brew install chromaprint``,
+``apt install libchromaprint-tools``, or https://acoustid.org/chromaprint),
+and no Python package.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import struct
+import subprocess
+from typing import Any, Sequence
+
+from .registry import ExtractionFailed, Extractor, register
+
+logger = logging.getLogger(__name__)
+
+_HEADER = struct.Struct("<dI")
+#: Fingerprint items two recordings may be shifted by (one item ≈ 0.124 s),
+#: for a copy with a little more or less silence at the start.
+_MAX_SHIFT = 12
+#: The overlap must cover this share of the shorter fingerprint.
+_MIN_OVERLAP = 0.5
+#: Durations further apart than this (seconds, or share of the longer) are
+#: different recordings; cheaper than comparing the fingerprints.
+_DURATION_SLACK_S = 3.0
+_DURATION_SLACK = 0.1
+_TIMEOUT = 120
+
+
+@register
+class AudioExtractor(Extractor):
+    """A Chromaprint fingerprint of the first ``length`` seconds, compared by bit error rate.
+
+    The fingerprints are aligned at the best of a few small shifts. The
+    similarity is ``1 - 2 * bit_error_rate``, clamped at 0: the same
+    recording re-encoded scores near 1, and unrelated audio (about half the
+    bits differ) near 0. Durations that differ by more than 3 s and 10%
+    score 0 without comparing.
+    """
+
+    name = "audio"
+    version = 1
+    extensions = frozenset({
+        ".mp3", ".m4a", ".aac", ".flac", ".wav", ".aif", ".aiff", ".ogg", ".oga",
+        ".opus", ".wma", ".alac", ".ape", ".wv",
+    })
+    requires_programs = ("fpcalc",)
+    needs_path = True
+    default_params = {"length": 120}
+    default_threshold = 0.7
+    description = "Similar audio: the same recording re-encoded or re-tagged (Chromaprint)"
+
+    def validate_params(self, params: dict[str, Any]) -> None:
+        if not 10 <= params["length"] <= 3600:
+            raise ValueError("length must be 10–3600 seconds")
+
+    def extract_file(self, path: str) -> bytes:
+        args = ["fpcalc", "-raw", "-json", "-length", str(self.params["length"]), path]
+        try:
+            proc = subprocess.run(args, stdin=subprocess.DEVNULL, capture_output=True, timeout=_TIMEOUT)
+        except subprocess.TimeoutExpired as exc:
+            raise ExtractionFailed(f"fpcalc took longer than {_TIMEOUT}s") from exc
+        except OSError as exc:
+            raise ExtractionFailed(f"cannot run fpcalc: {exc}") from exc
+        if proc.returncode != 0:
+            msg = proc.stderr.decode("utf-8", "replace").strip().splitlines()
+            raise ExtractionFailed(msg[-1] if msg else f"fpcalc exited with {proc.returncode}")
+        try:
+            info = json.loads(proc.stdout.decode("utf-8", "replace"))
+            duration = float(info["duration"])
+            fp = [int(v) & 0xFFFFFFFF for v in info["fingerprint"]]
+        except (ValueError, KeyError, TypeError) as exc:
+            raise ExtractionFailed(f"fpcalc gave no readable fingerprint: {exc}") from exc
+        if not fp:
+            raise ExtractionFailed("too short to fingerprint")
+        return _HEADER.pack(duration, len(fp)) + struct.pack(f"<{len(fp)}I", *fp)
+
+    @staticmethod
+    def _decode(feature: bytes) -> tuple[float, int, int]:
+        """``(duration, items, fingerprint as one integer)``: item ``i`` in bits ``32i…32i+31``."""
+        duration, n = _HEADER.unpack_from(feature)
+        return duration, n, int.from_bytes(feature[_HEADER.size:_HEADER.size + 4 * n], "little")
+
+    def similarity(self, a: bytes, b: bytes) -> float:
+        return _compare(self._decode(a), self._decode(b))
+
+    def prepare(self, features: Sequence[bytes]) -> Any:
+        return [self._decode(f) for f in features]
+
+    def similarities(self, prepared: Any, query: bytes) -> Sequence[float]:
+        q = self._decode(query)
+        return [_compare(q, p) for p in prepared]
+
+
+def _compare(a: tuple[float, int, int], b: tuple[float, int, int]) -> float:
+    (da, na, fa), (db, nb, fb) = a, b
+    if abs(da - db) > max(_DURATION_SLACK_S, _DURATION_SLACK * max(da, db)):
+        return 0.0
+    need = max(1, int(min(na, nb) * _MIN_OVERLAP))
+    best = 0.0
+    for shift in range(-_MAX_SHIFT, _MAX_SHIFT + 1):
+        # Item i of one lines up with item i + shift of the other.
+        x, y, nx, ny = (fa, fb, na, nb) if shift >= 0 else (fb, fa, nb, na)
+        k = abs(shift)
+        overlap = min(nx - k, ny)
+        if overlap < need:
+            continue
+        bits = 32 * overlap
+        mask = (1 << bits) - 1
+        errors = (((x >> (32 * k)) ^ y) & mask).bit_count()
+        best = max(best, 1.0 - 2.0 * errors / bits)
+    return best