imgsim-py 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (34) hide show
  1. imgsim_py-0.1.0/CHANGELOG.md +35 -0
  2. imgsim_py-0.1.0/CONTRIBUTING.md +63 -0
  3. imgsim_py-0.1.0/LICENSE +21 -0
  4. imgsim_py-0.1.0/MANIFEST.in +23 -0
  5. imgsim_py-0.1.0/PKG-INFO +249 -0
  6. imgsim_py-0.1.0/README.md +213 -0
  7. imgsim_py-0.1.0/README.zh-CN.md +198 -0
  8. imgsim_py-0.1.0/examples/quickstart.py +67 -0
  9. imgsim_py-0.1.0/pyproject.toml +87 -0
  10. imgsim_py-0.1.0/setup.cfg +4 -0
  11. imgsim_py-0.1.0/src/imgsim/__init__.py +49 -0
  12. imgsim_py-0.1.0/src/imgsim/__main__.py +6 -0
  13. imgsim_py-0.1.0/src/imgsim/_io.py +87 -0
  14. imgsim_py-0.1.0/src/imgsim/_types.py +103 -0
  15. imgsim_py-0.1.0/src/imgsim/cli.py +267 -0
  16. imgsim_py-0.1.0/src/imgsim/cluster.py +72 -0
  17. imgsim_py-0.1.0/src/imgsim/color.py +105 -0
  18. imgsim_py-0.1.0/src/imgsim/embed.py +106 -0
  19. imgsim_py-0.1.0/src/imgsim/fingerprint.py +133 -0
  20. imgsim_py-0.1.0/src/imgsim/index.py +527 -0
  21. imgsim_py-0.1.0/src/imgsim/py.typed +0 -0
  22. imgsim_py-0.1.0/src/imgsim/report.py +138 -0
  23. imgsim_py-0.1.0/src/imgsim_py.egg-info/PKG-INFO +249 -0
  24. imgsim_py-0.1.0/src/imgsim_py.egg-info/SOURCES.txt +32 -0
  25. imgsim_py-0.1.0/src/imgsim_py.egg-info/dependency_links.txt +1 -0
  26. imgsim_py-0.1.0/src/imgsim_py.egg-info/entry_points.txt +2 -0
  27. imgsim_py-0.1.0/src/imgsim_py.egg-info/requires.txt +11 -0
  28. imgsim_py-0.1.0/src/imgsim_py.egg-info/top_level.txt +1 -0
  29. imgsim_py-0.1.0/tests/conftest.py +116 -0
  30. imgsim_py-0.1.0/tests/test_cli.py +113 -0
  31. imgsim_py-0.1.0/tests/test_cluster.py +77 -0
  32. imgsim_py-0.1.0/tests/test_color.py +86 -0
  33. imgsim_py-0.1.0/tests/test_fingerprint.py +136 -0
  34. imgsim_py-0.1.0/tests/test_index.py +210 -0
@@ -0,0 +1,35 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project are documented here.
4
+ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
5
+ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
+
7
+ ## [Unreleased]
8
+
9
+ ## [0.1.0] - 2026-11-18
10
+
11
+ First public release.
12
+
13
+ ### Added
14
+
15
+ - `ImageIndex` with `add`, `add_many`, `add_dir`, `compare`, `nearest`, `save`, `load`,
16
+ `iter_pairs` and `clear`.
17
+ - Perceptual hashing: DCT `pHash` plus gradient `dHash`, averaged into one distance.
18
+ - Color veto in HSL space, weighted by the smaller of the two saturations so that
19
+ grayscale conversions are not rejected while differently-colored subjects are.
20
+ - Low-information guard: fingerprints on flat images (solid colors, line art, stripes)
21
+ are flagged by AC/DC energy ratio and require near-identical color to match.
22
+ - Union-find clustering, so similarity chains group transitively.
23
+ - CLI (`imgsim`) with text, JSON and CSV output, cache reuse, query mode,
24
+ top-k export and `--fail-on-duplicates` for CI.
25
+ - Self-contained HTML report with embedded thumbnails.
26
+ - Optional `embed` mode using ResNet18 features and cosine distance.
27
+
28
+ ### Performance
29
+
30
+ - 1000 images (400×400) indexed in 1.7 s and compared in 2.3 s on 4 cores.
31
+ - `Image.draft()` downsampling during JPEG decode, 5–10× on large photos.
32
+ - Hamming distance via `numpy.bitwise_count` hardware popcount.
33
+
34
+ [Unreleased]: https://github.com/zhouchuan315-png/imgsim/compare/v0.1.0...HEAD
35
+ [0.1.0]: https://github.com/zhouchuan315-png/imgsim/releases/tag/v0.1.0
@@ -0,0 +1,63 @@
1
+ # Contributing to imgsim
2
+
3
+ Thanks for taking the time to contribute. This is a small library, so the process is
4
+ short.
5
+
6
+ ## Setup
7
+
8
+ ```bash
9
+ git clone https://github.com/zhouchuan315-png/imgsim
10
+ cd imgsim
11
+ python -m venv .venv
12
+ source .venv/bin/activate # Windows: .venv\Scripts\activate
13
+ pip install -e ".[dev]"
14
+ ```
15
+
16
+ ## Before opening a PR
17
+
18
+ ```bash
19
+ pytest # all tests must pass
20
+ ruff check . # lint must be clean
21
+ ```
22
+
23
+ CI runs both on Linux, macOS and Windows for Python 3.9–3.13. If a test only fails on
24
+ one platform, say so in the PR — it is usually a path or encoding issue, not logic.
25
+
26
+ ## Guidelines
27
+
28
+ **Tests come with the change.** Every bug fix needs a regression test that fails
29
+ before the fix. Several tests in this repo are named after the failure they guard
30
+ against (`test_flat_white_and_stripes_are_separated`, `test_pack_bits_roundtrip_preserves_bit_order`)
31
+ — keep that convention.
32
+
33
+ **No binary fixtures.** Tests generate images from a fixed seed via
34
+ `tests/conftest.py::make_image`. The repo stays small and results stay reproducible.
35
+
36
+ **Changing the algorithm changes everyone's thresholds.** If you touch the hashing or
37
+ the color distance, include before/after numbers on a corpus of at least 100 images in
38
+ the PR description. A change that improves one case by 10% while degrading another by
39
+ 5% is usually not worth it.
40
+
41
+ **Keep the dependency footprint at two.** `numpy` and `Pillow`, nothing else in the
42
+ core path. Anything heavier goes behind an optional extra. This is why Hamming distance
43
+ has a numpy-1.x fallback instead of just requiring scipy.
44
+
45
+ ## Reporting bugs
46
+
47
+ Include:
48
+
49
+ - `imgsim` version, Python version, OS
50
+ - The command or code snippet you ran
51
+ - What you expected vs what happened
52
+ - If it is a misclassification: two example images (or a description that lets us
53
+ reproduce them) and the threshold you used
54
+
55
+ ## Requesting features
56
+
57
+ Open an issue describing the problem, not the solution. "Our photo library has 40k
58
+ images and clustering takes 8 minutes" is more useful than "add FAISS support" — the
59
+ right fix might be bucketing, a coarser hash, or something we haven't thought of.
60
+
61
+ ## License
62
+
63
+ By contributing you agree that your contribution is licensed under the MIT License.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 zhou
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,23 @@
1
+ # Files to include in the source distribution beyond what setuptools discovers.
2
+ # The wheel only ever contains src/imgsim; everything here is for people who
3
+ # download the sdist to build, test or read.
4
+
5
+ include LICENSE
6
+ include README.md
7
+ include README.zh-CN.md
8
+ include CHANGELOG.md
9
+ include CONTRIBUTING.md
10
+ include pyproject.toml
11
+
12
+ # setuptools only auto-includes tests/test*.py, which silently drops conftest.py.
13
+ # Without it the shipped test suite cannot run, so include the whole directory.
14
+ recursive-include tests *.py
15
+ recursive-include examples *.py
16
+
17
+ # Never ship local build or cache artefacts.
18
+ # setuptools already excludes __pycache__ and *.pyc by default; these prunes handle
19
+ # the directories that would otherwise be picked up when building in-place.
20
+ prune .github
21
+ prune _distcheck
22
+ prune dist
23
+ prune build
@@ -0,0 +1,249 @@
1
+ Metadata-Version: 2.4
2
+ Name: imgsim-py
3
+ Version: 0.1.0
4
+ Summary: Find duplicate and near-duplicate images with perceptual hashing, color verification and clustering.
5
+ Author-email: zhou <zhouchuan315@gmail.com>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/zhouchuan315-png/imgsim
8
+ Project-URL: Repository, https://github.com/zhouchuan315-png/imgsim
9
+ Project-URL: Issues, https://github.com/zhouchuan315-png/imgsim/issues
10
+ Project-URL: Changelog, https://github.com/zhouchuan315-png/imgsim/blob/main/CHANGELOG.md
11
+ Keywords: image,similarity,duplicate,deduplication,perceptual-hash,phash,near-duplicate,image-dedup
12
+ Classifier: Development Status :: 4 - Beta
13
+ Classifier: Intended Audience :: Developers
14
+ Classifier: Operating System :: OS Independent
15
+ Classifier: Programming Language :: Python :: 3
16
+ Classifier: Programming Language :: Python :: 3.9
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Topic :: Multimedia :: Graphics
22
+ Classifier: Typing :: Typed
23
+ Requires-Python: >=3.9
24
+ Description-Content-Type: text/markdown
25
+ License-File: LICENSE
26
+ Requires-Dist: numpy>=1.21
27
+ Requires-Dist: Pillow>=9.0
28
+ Provides-Extra: embed
29
+ Requires-Dist: torch>=2.0; extra == "embed"
30
+ Requires-Dist: torchvision>=0.15; extra == "embed"
31
+ Provides-Extra: dev
32
+ Requires-Dist: pytest>=7.4; extra == "dev"
33
+ Requires-Dist: pytest-cov>=4.1; extra == "dev"
34
+ Requires-Dist: ruff>=0.4; extra == "dev"
35
+ Dynamic: license-file
36
+
37
+ # imgsim
38
+
39
+ **Find duplicate and near-duplicate images in a folder — and actually get the right answer.**
40
+
41
+ [![CI](https://github.com/zhouchuan315-png/imgsim/actions/workflows/ci.yml/badge.svg)](https://github.com/zhouchuan315-png/imgsim/actions/workflows/ci.yml)
42
+ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
43
+ [![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](pyproject.toml)
44
+ [![PyPI](https://img.shields.io/pypi/v/imgsim-py.svg)](https://pypi.org/project/imgsim-py/)
45
+
46
+ `imgsim` indexes a directory of images, computes a 64-bit perceptual fingerprint for
47
+ each one, and groups the ones that are the same picture — resized, recompressed,
48
+ cropped, brightened, or converted to black and white.
49
+
50
+ It exists because plain perceptual hashing has two failure modes that show up in
51
+ every real photo library:
52
+
53
+ | Failure | What you see without a fix |
54
+ |---|---|
55
+ | **Color blindness.** pHash works on grayscale. | A red apple and a green apple of the same shape are reported identical. |
56
+ | **Fingerprint collapse.** Flat images have no low-frequency structure. | A white rectangle and a blue-white stripe pattern hash to the same all-zero fingerprint. |
57
+
58
+ `imgsim` adds a hue-aware color veto and a low-information guard on top of the
59
+ standard pHash + dHash pair. On the bundled benchmark (1000 images, 20 families of
60
+ 50 variants each) it recovers **20/20 groups with zero cross-family contamination**.
61
+
62
+ ---
63
+
64
+ ## Install
65
+
66
+ ```bash
67
+ pip install imgsim-py
68
+ ```
69
+
70
+ The distribution is named `imgsim-py` (the plain `imgsim` name on PyPI belongs to an
71
+ unrelated, unmaintained project), but the import name is `imgsim`:
72
+
73
+ ```python
74
+ import imgsim
75
+ ```
76
+
77
+ Optional semantic mode:
78
+
79
+ ```bash
80
+ pip install imgsim-py[embed] # pulls PyTorch + torchvision (~2 GB)
81
+ ```
82
+
83
+ ## Quick start
84
+
85
+ ### Command line
86
+
87
+ ```bash
88
+ imgsim ./photos # group duplicates, write ./imgsim-out/report.html
89
+ imgsim ./photos -r -t 8 # recurse, stricter threshold
90
+ imgsim ./photos --query cat.jpg -k 5 # what else looks like cat.jpg?
91
+ imgsim ./photos -f json | jq '.groups | length'
92
+ ```
93
+
94
+ `imgsim --help` for everything.
95
+
96
+ ### Python
97
+
98
+ ```python
99
+ import imgsim
100
+
101
+ index = imgsim.ImageIndex()
102
+ index.add_dir("photos/", recursive=True) # decodes once, in parallel
103
+
104
+ result = index.compare(threshold=10) # Hamming bits, out of 64
105
+ for group in result.groups:
106
+ print(len(group), "similar images")
107
+ for path, distance in zip(group.paths, group.distances):
108
+ print(f" {distance:>6.1f} {path}")
109
+
110
+ imgsim.save_html(result, "report.html")
111
+ ```
112
+
113
+ Fingerprints are cheap to keep around, so threshold tuning is instant:
114
+
115
+ ```python
116
+ index.save("photos.imgsim") # ~300 bytes per image
117
+ index = imgsim.ImageIndex.load("photos.imgsim")
118
+ result = index.compare(threshold=6) # no re-decoding
119
+ ```
120
+
121
+ Nearest-neighbour lookup for an image that is not in the index:
122
+
123
+ ```python
124
+ for path, distance in index.nearest("new-photo.jpg", k=5):
125
+ print(distance, path)
126
+ ```
127
+
128
+ ## Choosing a threshold
129
+
130
+ `threshold` is the maximum Hamming distance between two 64-bit fingerprints —
131
+ literally *how many of the 64 bits are allowed to differ*.
132
+
133
+ | Value | What it finds |
134
+ |---|---|
135
+ | 0–4 | Byte-identical copies, re-encodes, metadata-only edits |
136
+ | **10 (default)** | Resized, recompressed, brightness-shifted, grayscale versions |
137
+ | 14–18 | The same asset cropped or slightly recomposed |
138
+ | >20 | Starts merging genuinely different photos |
139
+
140
+ Screenshots and digital art usually want 6–8. Phone photo libraries need 10–14
141
+ because of sensor noise and HDR processing. **Calibrate on a 200-image sample
142
+ first** — the right value varies by roughly 2x between sources.
143
+
144
+ ## How it works
145
+
146
+ Four steps per image, all numpy, no OpenCV:
147
+
148
+ 1. **Resize to 32×32 grayscale.** Kills resolution and compression differences.
149
+ 2. **2D DCT, keep the top-left 8×8.** Low frequencies describe layout, not pixels.
150
+ 3. **Compare each coefficient to the block median.** Above → 1, below → 0. That's 64 bits.
151
+ 4. **Distance = popcount of the XOR.** Two fingerprints, one XOR, count the 1s.
152
+
153
+ Two hashes are computed and their distances averaged: `pHash` (DCT, survives resize
154
+ and recompression) and `dHash` (horizontal gradients, sensitive to real structural
155
+ edits). Averaging suppresses most false positives that either would produce alone.
156
+
157
+ ### The color veto
158
+
159
+ A 192-float signature is taken by averaging an 8×8 RGB grid, then compared in HSL:
160
+
161
+ ```
162
+ distance = 0.55·Δhue·minSat + 0.35·Δsat·minSat + 0.90·Δlightness
163
+ ```
164
+
165
+ The `minSat` factor is the important part. Grayscale conversion destroys saturation
166
+ but not perceived lightness, so weighting hue and saturation by the *smaller* of the
167
+ two saturations makes those terms vanish for black-and-white images — only lightness
168
+ is compared. "Photo vs its B&W version" passes; "red apple vs green apple" (both
169
+ saturated) is rejected at full strength.
170
+
171
+ Only the most-different 12 of the 64 cells are averaged, so recoloring one object in
172
+ a corner is not diluted away by 52 unchanged cells.
173
+
174
+ Measured separation: same-image variants score **0.002–0.06**, unrelated images score
175
+ **0.39–0.90**. The default cutoff of 0.18 sits in the gap.
176
+
177
+ ### The low-information guard
178
+
179
+ A pHash is only meaningful if the image has low-frequency structure. For solid
180
+ colors, line art, and regular stripes, the AC coefficients are ~0, the median
181
+ comparison degenerates, and every such image hashes identically. `imgsim` measures
182
+ the AC/DC energy ratio and, below 0.02, demands near-identical color (0.06) before
183
+ calling two images similar.
184
+
185
+ ## Performance
186
+
187
+ Measured on a 4-core Windows laptop, 1000 JPEGs at 400×400:
188
+
189
+ | Stage | Time |
190
+ |---|---|
191
+ | Decode + fingerprint | 1.7 s |
192
+ | Similarity matrix + clustering | 2.3 s |
193
+ | **Total** | **4.0 s** |
194
+
195
+ Two things make this fast: `Image.draft()` lets libjpeg downsample during decoding
196
+ (5–10× on multi-megapixel photos), and Hamming distance runs through
197
+ `numpy.bitwise_count`, a hardware popcount, over uint64 codes.
198
+
199
+ Memory is a fingerprint per image — 8 bytes of hash plus a 192-float color signature
200
+ — so a million images fit comfortably in RAM.
201
+
202
+ Beyond ~50k images the dense `N×N` matrix becomes the bottleneck (50k → 2.5 GB).
203
+ At that point use `index.iter_pairs()` to stream instead, or bucket by hash prefix
204
+ before comparing. The API leaves room for both.
205
+
206
+ ## Semantic mode
207
+
208
+ Perceptual hashing answers *"is this the same picture?"*. It is blind to meaning: two
209
+ photos of the same beach from different angles hash completely differently.
210
+
211
+ ```bash
212
+ imgsim ./photos --mode embed --threshold 0.15
213
+ ```
214
+
215
+ `embed` mode extracts 512-d ResNet18 features and compares cosine distance. It
216
+ recognises the same subject across viewpoints, at the cost of a PyTorch dependency
217
+ and much weaker fine-grained discrimination — it will happily merge two different
218
+ sunsets. Use it when "similar subject" matters more than "same file".
219
+
220
+ ## API at a glance
221
+
222
+ | Member | Purpose |
223
+ |---|---|
224
+ | `ImageIndex(mode="hash")` | Build an index |
225
+ | `.add(path)` / `.add_many(paths)` / `.add_dir(dir)` | Index images (parallel decode) |
226
+ | `.compare(threshold, color_threshold, pairs_limit)` | Group into `Result` |
227
+ | `.nearest(path, k)` | Top-k neighbours of any image |
228
+ | `.save(path)` / `ImageIndex.load(path)` | Persist / restore fingerprints |
229
+ | `.iter_pairs(threshold)` | Stream pairs without materialising them |
230
+ | `.failed` | Paths that could not be decoded |
231
+ | `Result.groups` / `.pairs` / `.duplicates` | Results |
232
+ | `render_html(result)` / `save_html(result, path)` | Report |
233
+
234
+ ## Development
235
+
236
+ ```bash
237
+ git clone https://github.com/zhouchuan315-png/imgsim
238
+ cd imgsim
239
+ pip install -e ".[dev]"
240
+ pytest
241
+ ```
242
+
243
+ Tests build their images from a fixed seed — no binary fixtures in the repo.
244
+
245
+ ## License
246
+
247
+ MIT. See [LICENSE](LICENSE).
248
+
249
+ 中文文档:[README.zh-CN.md](README.zh-CN.md)
@@ -0,0 +1,213 @@
1
+ # imgsim
2
+
3
+ **Find duplicate and near-duplicate images in a folder — and actually get the right answer.**
4
+
5
+ [![CI](https://github.com/zhouchuan315-png/imgsim/actions/workflows/ci.yml/badge.svg)](https://github.com/zhouchuan315-png/imgsim/actions/workflows/ci.yml)
6
+ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
7
+ [![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](pyproject.toml)
8
+ [![PyPI](https://img.shields.io/pypi/v/imgsim-py.svg)](https://pypi.org/project/imgsim-py/)
9
+
10
+ `imgsim` indexes a directory of images, computes a 64-bit perceptual fingerprint for
11
+ each one, and groups the ones that are the same picture — resized, recompressed,
12
+ cropped, brightened, or converted to black and white.
13
+
14
+ It exists because plain perceptual hashing has two failure modes that show up in
15
+ every real photo library:
16
+
17
+ | Failure | What you see without a fix |
18
+ |---|---|
19
+ | **Color blindness.** pHash works on grayscale. | A red apple and a green apple of the same shape are reported identical. |
20
+ | **Fingerprint collapse.** Flat images have no low-frequency structure. | A white rectangle and a blue-white stripe pattern hash to the same all-zero fingerprint. |
21
+
22
+ `imgsim` adds a hue-aware color veto and a low-information guard on top of the
23
+ standard pHash + dHash pair. On the bundled benchmark (1000 images, 20 families of
24
+ 50 variants each) it recovers **20/20 groups with zero cross-family contamination**.
25
+
26
+ ---
27
+
28
+ ## Install
29
+
30
+ ```bash
31
+ pip install imgsim-py
32
+ ```
33
+
34
+ The distribution is named `imgsim-py` (the plain `imgsim` name on PyPI belongs to an
35
+ unrelated, unmaintained project), but the import name is `imgsim`:
36
+
37
+ ```python
38
+ import imgsim
39
+ ```
40
+
41
+ Optional semantic mode:
42
+
43
+ ```bash
44
+ pip install imgsim-py[embed] # pulls PyTorch + torchvision (~2 GB)
45
+ ```
46
+
47
+ ## Quick start
48
+
49
+ ### Command line
50
+
51
+ ```bash
52
+ imgsim ./photos # group duplicates, write ./imgsim-out/report.html
53
+ imgsim ./photos -r -t 8 # recurse, stricter threshold
54
+ imgsim ./photos --query cat.jpg -k 5 # what else looks like cat.jpg?
55
+ imgsim ./photos -f json | jq '.groups | length'
56
+ ```
57
+
58
+ `imgsim --help` for everything.
59
+
60
+ ### Python
61
+
62
+ ```python
63
+ import imgsim
64
+
65
+ index = imgsim.ImageIndex()
66
+ index.add_dir("photos/", recursive=True) # decodes once, in parallel
67
+
68
+ result = index.compare(threshold=10) # Hamming bits, out of 64
69
+ for group in result.groups:
70
+ print(len(group), "similar images")
71
+ for path, distance in zip(group.paths, group.distances):
72
+ print(f" {distance:>6.1f} {path}")
73
+
74
+ imgsim.save_html(result, "report.html")
75
+ ```
76
+
77
+ Fingerprints are cheap to keep around, so threshold tuning is instant:
78
+
79
+ ```python
80
+ index.save("photos.imgsim") # ~300 bytes per image
81
+ index = imgsim.ImageIndex.load("photos.imgsim")
82
+ result = index.compare(threshold=6) # no re-decoding
83
+ ```
84
+
85
+ Nearest-neighbour lookup for an image that is not in the index:
86
+
87
+ ```python
88
+ for path, distance in index.nearest("new-photo.jpg", k=5):
89
+ print(distance, path)
90
+ ```
91
+
92
+ ## Choosing a threshold
93
+
94
+ `threshold` is the maximum Hamming distance between two 64-bit fingerprints —
95
+ literally *how many of the 64 bits are allowed to differ*.
96
+
97
+ | Value | What it finds |
98
+ |---|---|
99
+ | 0–4 | Byte-identical copies, re-encodes, metadata-only edits |
100
+ | **10 (default)** | Resized, recompressed, brightness-shifted, grayscale versions |
101
+ | 14–18 | The same asset cropped or slightly recomposed |
102
+ | >20 | Starts merging genuinely different photos |
103
+
104
+ Screenshots and digital art usually want 6–8. Phone photo libraries need 10–14
105
+ because of sensor noise and HDR processing. **Calibrate on a 200-image sample
106
+ first** — the right value varies by roughly 2x between sources.
107
+
108
+ ## How it works
109
+
110
+ Four steps per image, all numpy, no OpenCV:
111
+
112
+ 1. **Resize to 32×32 grayscale.** Kills resolution and compression differences.
113
+ 2. **2D DCT, keep the top-left 8×8.** Low frequencies describe layout, not pixels.
114
+ 3. **Compare each coefficient to the block median.** Above → 1, below → 0. That's 64 bits.
115
+ 4. **Distance = popcount of the XOR.** Two fingerprints, one XOR, count the 1s.
116
+
117
+ Two hashes are computed and their distances averaged: `pHash` (DCT, survives resize
118
+ and recompression) and `dHash` (horizontal gradients, sensitive to real structural
119
+ edits). Averaging suppresses most false positives that either would produce alone.
120
+
121
+ ### The color veto
122
+
123
+ A 192-float signature is taken by averaging an 8×8 RGB grid, then compared in HSL:
124
+
125
+ ```
126
+ distance = 0.55·Δhue·minSat + 0.35·Δsat·minSat + 0.90·Δlightness
127
+ ```
128
+
129
+ The `minSat` factor is the important part. Grayscale conversion destroys saturation
130
+ but not perceived lightness, so weighting hue and saturation by the *smaller* of the
131
+ two saturations makes those terms vanish for black-and-white images — only lightness
132
+ is compared. "Photo vs its B&W version" passes; "red apple vs green apple" (both
133
+ saturated) is rejected at full strength.
134
+
135
+ Only the most-different 12 of the 64 cells are averaged, so recoloring one object in
136
+ a corner is not diluted away by 52 unchanged cells.
137
+
138
+ Measured separation: same-image variants score **0.002–0.06**, unrelated images score
139
+ **0.39–0.90**. The default cutoff of 0.18 sits in the gap.
140
+
141
+ ### The low-information guard
142
+
143
+ A pHash is only meaningful if the image has low-frequency structure. For solid
144
+ colors, line art, and regular stripes, the AC coefficients are ~0, the median
145
+ comparison degenerates, and every such image hashes identically. `imgsim` measures
146
+ the AC/DC energy ratio and, below 0.02, demands near-identical color (0.06) before
147
+ calling two images similar.
148
+
149
+ ## Performance
150
+
151
+ Measured on a 4-core Windows laptop, 1000 JPEGs at 400×400:
152
+
153
+ | Stage | Time |
154
+ |---|---|
155
+ | Decode + fingerprint | 1.7 s |
156
+ | Similarity matrix + clustering | 2.3 s |
157
+ | **Total** | **4.0 s** |
158
+
159
+ Two things make this fast: `Image.draft()` lets libjpeg downsample during decoding
160
+ (5–10× on multi-megapixel photos), and Hamming distance runs through
161
+ `numpy.bitwise_count`, a hardware popcount, over uint64 codes.
162
+
163
+ Memory is a fingerprint per image — 8 bytes of hash plus a 192-float color signature
164
+ — so a million images fit comfortably in RAM.
165
+
166
+ Beyond ~50k images the dense `N×N` matrix becomes the bottleneck (50k → 2.5 GB).
167
+ At that point use `index.iter_pairs()` to stream instead, or bucket by hash prefix
168
+ before comparing. The API leaves room for both.
169
+
170
+ ## Semantic mode
171
+
172
+ Perceptual hashing answers *"is this the same picture?"*. It is blind to meaning: two
173
+ photos of the same beach from different angles hash completely differently.
174
+
175
+ ```bash
176
+ imgsim ./photos --mode embed --threshold 0.15
177
+ ```
178
+
179
+ `embed` mode extracts 512-d ResNet18 features and compares cosine distance. It
180
+ recognises the same subject across viewpoints, at the cost of a PyTorch dependency
181
+ and much weaker fine-grained discrimination — it will happily merge two different
182
+ sunsets. Use it when "similar subject" matters more than "same file".
183
+
184
+ ## API at a glance
185
+
186
+ | Member | Purpose |
187
+ |---|---|
188
+ | `ImageIndex(mode="hash")` | Build an index |
189
+ | `.add(path)` / `.add_many(paths)` / `.add_dir(dir)` | Index images (parallel decode) |
190
+ | `.compare(threshold, color_threshold, pairs_limit)` | Group into `Result` |
191
+ | `.nearest(path, k)` | Top-k neighbours of any image |
192
+ | `.save(path)` / `ImageIndex.load(path)` | Persist / restore fingerprints |
193
+ | `.iter_pairs(threshold)` | Stream pairs without materialising them |
194
+ | `.failed` | Paths that could not be decoded |
195
+ | `Result.groups` / `.pairs` / `.duplicates` | Results |
196
+ | `render_html(result)` / `save_html(result, path)` | Report |
197
+
198
+ ## Development
199
+
200
+ ```bash
201
+ git clone https://github.com/zhouchuan315-png/imgsim
202
+ cd imgsim
203
+ pip install -e ".[dev]"
204
+ pytest
205
+ ```
206
+
207
+ Tests build their images from a fixed seed — no binary fixtures in the repo.
208
+
209
+ ## License
210
+
211
+ MIT. See [LICENSE](LICENSE).
212
+
213
+ 中文文档:[README.zh-CN.md](README.zh-CN.md)