imgsim-py 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- imgsim_py-0.1.0/CHANGELOG.md +35 -0
- imgsim_py-0.1.0/CONTRIBUTING.md +63 -0
- imgsim_py-0.1.0/LICENSE +21 -0
- imgsim_py-0.1.0/MANIFEST.in +23 -0
- imgsim_py-0.1.0/PKG-INFO +249 -0
- imgsim_py-0.1.0/README.md +213 -0
- imgsim_py-0.1.0/README.zh-CN.md +198 -0
- imgsim_py-0.1.0/examples/quickstart.py +67 -0
- imgsim_py-0.1.0/pyproject.toml +87 -0
- imgsim_py-0.1.0/setup.cfg +4 -0
- imgsim_py-0.1.0/src/imgsim/__init__.py +49 -0
- imgsim_py-0.1.0/src/imgsim/__main__.py +6 -0
- imgsim_py-0.1.0/src/imgsim/_io.py +87 -0
- imgsim_py-0.1.0/src/imgsim/_types.py +103 -0
- imgsim_py-0.1.0/src/imgsim/cli.py +267 -0
- imgsim_py-0.1.0/src/imgsim/cluster.py +72 -0
- imgsim_py-0.1.0/src/imgsim/color.py +105 -0
- imgsim_py-0.1.0/src/imgsim/embed.py +106 -0
- imgsim_py-0.1.0/src/imgsim/fingerprint.py +133 -0
- imgsim_py-0.1.0/src/imgsim/index.py +527 -0
- imgsim_py-0.1.0/src/imgsim/py.typed +0 -0
- imgsim_py-0.1.0/src/imgsim/report.py +138 -0
- imgsim_py-0.1.0/src/imgsim_py.egg-info/PKG-INFO +249 -0
- imgsim_py-0.1.0/src/imgsim_py.egg-info/SOURCES.txt +32 -0
- imgsim_py-0.1.0/src/imgsim_py.egg-info/dependency_links.txt +1 -0
- imgsim_py-0.1.0/src/imgsim_py.egg-info/entry_points.txt +2 -0
- imgsim_py-0.1.0/src/imgsim_py.egg-info/requires.txt +11 -0
- imgsim_py-0.1.0/src/imgsim_py.egg-info/top_level.txt +1 -0
- imgsim_py-0.1.0/tests/conftest.py +116 -0
- imgsim_py-0.1.0/tests/test_cli.py +113 -0
- imgsim_py-0.1.0/tests/test_cluster.py +77 -0
- imgsim_py-0.1.0/tests/test_color.py +86 -0
- imgsim_py-0.1.0/tests/test_fingerprint.py +136 -0
- imgsim_py-0.1.0/tests/test_index.py +210 -0
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here.
|
|
4
|
+
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
|
|
5
|
+
project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.1.0] - 2026-11-18
|
|
10
|
+
|
|
11
|
+
First public release.
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- `ImageIndex` with `add`, `add_many`, `add_dir`, `compare`, `nearest`, `save`, `load`,
|
|
16
|
+
`iter_pairs` and `clear`.
|
|
17
|
+
- Perceptual hashing: DCT `pHash` plus gradient `dHash`, averaged into one distance.
|
|
18
|
+
- Color veto in HSL space, weighted by the smaller of the two saturations so that
|
|
19
|
+
grayscale conversions are not rejected while differently-colored subjects are.
|
|
20
|
+
- Low-information guard: fingerprints on flat images (solid colors, line art, stripes)
|
|
21
|
+
are flagged by AC/DC energy ratio and require near-identical color to match.
|
|
22
|
+
- Union-find clustering, so similarity chains group transitively.
|
|
23
|
+
- CLI (`imgsim`) with text, JSON and CSV output, cache reuse, query mode,
|
|
24
|
+
top-k export and `--fail-on-duplicates` for CI.
|
|
25
|
+
- Self-contained HTML report with embedded thumbnails.
|
|
26
|
+
- Optional `embed` mode using ResNet18 features and cosine distance.
|
|
27
|
+
|
|
28
|
+
### Performance
|
|
29
|
+
|
|
30
|
+
- 1000 images (400×400) indexed in 1.7 s and compared in 2.3 s on 4 cores.
|
|
31
|
+
- `Image.draft()` downsampling during JPEG decode, 5–10× on large photos.
|
|
32
|
+
- Hamming distance via `numpy.bitwise_count` hardware popcount.
|
|
33
|
+
|
|
34
|
+
[Unreleased]: https://github.com/zhouchuan315-png/imgsim/compare/v0.1.0...HEAD
|
|
35
|
+
[0.1.0]: https://github.com/zhouchuan315-png/imgsim/releases/tag/v0.1.0
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# Contributing to imgsim
|
|
2
|
+
|
|
3
|
+
Thanks for taking the time to contribute. This is a small library, so the process is
|
|
4
|
+
short.
|
|
5
|
+
|
|
6
|
+
## Setup
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
git clone https://github.com/zhouchuan315-png/imgsim
|
|
10
|
+
cd imgsim
|
|
11
|
+
python -m venv .venv
|
|
12
|
+
source .venv/bin/activate # Windows: .venv\Scripts\activate
|
|
13
|
+
pip install -e ".[dev]"
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
## Before opening a PR
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
pytest # all tests must pass
|
|
20
|
+
ruff check . # lint must be clean
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
CI runs both on Linux, macOS and Windows for Python 3.9–3.13. If a test only fails on
|
|
24
|
+
one platform, say so in the PR — it is usually a path or encoding issue, not logic.
|
|
25
|
+
|
|
26
|
+
## Guidelines
|
|
27
|
+
|
|
28
|
+
**Tests come with the change.** Every bug fix needs a regression test that fails
|
|
29
|
+
before the fix. Several tests in this repo are named after the failure they guard
|
|
30
|
+
against (`test_flat_white_and_stripes_are_separated`, `test_pack_bits_roundtrip_preserves_bit_order`)
|
|
31
|
+
— keep that convention.
|
|
32
|
+
|
|
33
|
+
**No binary fixtures.** Tests generate images from a fixed seed via
|
|
34
|
+
`tests/conftest.py::make_image`. The repo stays small and results stay reproducible.
|
|
35
|
+
|
|
36
|
+
**Changing the algorithm changes everyone's thresholds.** If you touch the hashing or
|
|
37
|
+
the color distance, include before/after numbers on a corpus of at least 100 images in
|
|
38
|
+
the PR description. A change that improves one case by 10% while degrading another by
|
|
39
|
+
5% is usually not worth it.
|
|
40
|
+
|
|
41
|
+
**Keep the dependency footprint at two.** `numpy` and `Pillow`, nothing else in the
|
|
42
|
+
core path. Anything heavier goes behind an optional extra. This is why Hamming distance
|
|
43
|
+
has a numpy-1.x fallback instead of just requiring scipy.
|
|
44
|
+
|
|
45
|
+
## Reporting bugs
|
|
46
|
+
|
|
47
|
+
Include:
|
|
48
|
+
|
|
49
|
+
- `imgsim` version, Python version, OS
|
|
50
|
+
- The command or code snippet you ran
|
|
51
|
+
- What you expected vs what happened
|
|
52
|
+
- If it is a misclassification: two example images (or a description that lets us
|
|
53
|
+
reproduce them) and the threshold you used
|
|
54
|
+
|
|
55
|
+
## Requesting features
|
|
56
|
+
|
|
57
|
+
Open an issue describing the problem, not the solution. "Our photo library has 40k
|
|
58
|
+
images and clustering takes 8 minutes" is more useful than "add FAISS support" — the
|
|
59
|
+
right fix might be bucketing, a coarser hash, or something we haven't thought of.
|
|
60
|
+
|
|
61
|
+
## License
|
|
62
|
+
|
|
63
|
+
By contributing you agree that your contribution is licensed under the MIT License.
|
imgsim_py-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 zhou
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Files to include in the source distribution beyond what setuptools discovers.
|
|
2
|
+
# The wheel only ever contains src/imgsim; everything here is for people who
|
|
3
|
+
# download the sdist to build, test or read.
|
|
4
|
+
|
|
5
|
+
include LICENSE
|
|
6
|
+
include README.md
|
|
7
|
+
include README.zh-CN.md
|
|
8
|
+
include CHANGELOG.md
|
|
9
|
+
include CONTRIBUTING.md
|
|
10
|
+
include pyproject.toml
|
|
11
|
+
|
|
12
|
+
# setuptools only auto-includes tests/test*.py, which silently drops conftest.py.
|
|
13
|
+
# Without it the shipped test suite cannot run, so include the whole directory.
|
|
14
|
+
recursive-include tests *.py
|
|
15
|
+
recursive-include examples *.py
|
|
16
|
+
|
|
17
|
+
# Never ship local build or cache artefacts.
|
|
18
|
+
# setuptools already excludes __pycache__ and *.pyc by default; these prunes handle
|
|
19
|
+
# the directories that would otherwise be picked up when building in-place.
|
|
20
|
+
prune .github
|
|
21
|
+
prune _distcheck
|
|
22
|
+
prune dist
|
|
23
|
+
prune build
|
imgsim_py-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,249 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: imgsim-py
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Find duplicate and near-duplicate images with perceptual hashing, color verification and clustering.
|
|
5
|
+
Author-email: zhou <zhouchuan315@gmail.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/zhouchuan315-png/imgsim
|
|
8
|
+
Project-URL: Repository, https://github.com/zhouchuan315-png/imgsim
|
|
9
|
+
Project-URL: Issues, https://github.com/zhouchuan315-png/imgsim/issues
|
|
10
|
+
Project-URL: Changelog, https://github.com/zhouchuan315-png/imgsim/blob/main/CHANGELOG.md
|
|
11
|
+
Keywords: image,similarity,duplicate,deduplication,perceptual-hash,phash,near-duplicate,image-dedup
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Topic :: Multimedia :: Graphics
|
|
22
|
+
Classifier: Typing :: Typed
|
|
23
|
+
Requires-Python: >=3.9
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
License-File: LICENSE
|
|
26
|
+
Requires-Dist: numpy>=1.21
|
|
27
|
+
Requires-Dist: Pillow>=9.0
|
|
28
|
+
Provides-Extra: embed
|
|
29
|
+
Requires-Dist: torch>=2.0; extra == "embed"
|
|
30
|
+
Requires-Dist: torchvision>=0.15; extra == "embed"
|
|
31
|
+
Provides-Extra: dev
|
|
32
|
+
Requires-Dist: pytest>=7.4; extra == "dev"
|
|
33
|
+
Requires-Dist: pytest-cov>=4.1; extra == "dev"
|
|
34
|
+
Requires-Dist: ruff>=0.4; extra == "dev"
|
|
35
|
+
Dynamic: license-file
|
|
36
|
+
|
|
37
|
+
# imgsim
|
|
38
|
+
|
|
39
|
+
**Find duplicate and near-duplicate images in a folder — and actually get the right answer.**
|
|
40
|
+
|
|
41
|
+
[](https://github.com/zhouchuan315-png/imgsim/actions/workflows/ci.yml)
|
|
42
|
+
[](LICENSE)
|
|
43
|
+
[](pyproject.toml)
|
|
44
|
+
[](https://pypi.org/project/imgsim-py/)
|
|
45
|
+
|
|
46
|
+
`imgsim` indexes a directory of images, computes a 64-bit perceptual fingerprint for
|
|
47
|
+
each one, and groups the ones that are the same picture — resized, recompressed,
|
|
48
|
+
cropped, brightened, or converted to black and white.
|
|
49
|
+
|
|
50
|
+
It exists because plain perceptual hashing has two failure modes that show up in
|
|
51
|
+
every real photo library:
|
|
52
|
+
|
|
53
|
+
| Failure | What you see without a fix |
|
|
54
|
+
|---|---|
|
|
55
|
+
| **Color blindness.** pHash works on grayscale. | A red apple and a green apple of the same shape are reported identical. |
|
|
56
|
+
| **Fingerprint collapse.** Flat images have no low-frequency structure. | A white rectangle and a blue-white stripe pattern hash to the same all-zero fingerprint. |
|
|
57
|
+
|
|
58
|
+
`imgsim` adds a hue-aware color veto and a low-information guard on top of the
|
|
59
|
+
standard pHash + dHash pair. On the bundled benchmark (1000 images, 20 families of
|
|
60
|
+
50 variants each) it recovers **20/20 groups with zero cross-family contamination**.
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## Install
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
pip install imgsim-py
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
The distribution is named `imgsim-py` (the plain `imgsim` name on PyPI belongs to an
|
|
71
|
+
unrelated, unmaintained project), but the import name is `imgsim`:
|
|
72
|
+
|
|
73
|
+
```python
|
|
74
|
+
import imgsim
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Optional semantic mode:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
pip install imgsim-py[embed] # pulls PyTorch + torchvision (~2 GB)
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
## Quick start
|
|
84
|
+
|
|
85
|
+
### Command line
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
imgsim ./photos # group duplicates, write ./imgsim-out/report.html
|
|
89
|
+
imgsim ./photos -r -t 8 # recurse, stricter threshold
|
|
90
|
+
imgsim ./photos --query cat.jpg -k 5 # what else looks like cat.jpg?
|
|
91
|
+
imgsim ./photos -f json | jq '.groups | length'
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
`imgsim --help` for everything.
|
|
95
|
+
|
|
96
|
+
### Python
|
|
97
|
+
|
|
98
|
+
```python
|
|
99
|
+
import imgsim
|
|
100
|
+
|
|
101
|
+
index = imgsim.ImageIndex()
|
|
102
|
+
index.add_dir("photos/", recursive=True) # decodes once, in parallel
|
|
103
|
+
|
|
104
|
+
result = index.compare(threshold=10) # Hamming bits, out of 64
|
|
105
|
+
for group in result.groups:
|
|
106
|
+
print(len(group), "similar images")
|
|
107
|
+
for path, distance in zip(group.paths, group.distances):
|
|
108
|
+
print(f" {distance:>6.1f} {path}")
|
|
109
|
+
|
|
110
|
+
imgsim.save_html(result, "report.html")
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Fingerprints are cheap to keep around, so threshold tuning is instant:
|
|
114
|
+
|
|
115
|
+
```python
|
|
116
|
+
index.save("photos.imgsim") # ~300 bytes per image
|
|
117
|
+
index = imgsim.ImageIndex.load("photos.imgsim")
|
|
118
|
+
result = index.compare(threshold=6) # no re-decoding
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
Nearest-neighbour lookup for an image that is not in the index:
|
|
122
|
+
|
|
123
|
+
```python
|
|
124
|
+
for path, distance in index.nearest("new-photo.jpg", k=5):
|
|
125
|
+
print(distance, path)
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
## Choosing a threshold
|
|
129
|
+
|
|
130
|
+
`threshold` is the maximum Hamming distance between two 64-bit fingerprints —
|
|
131
|
+
literally *how many of the 64 bits are allowed to differ*.
|
|
132
|
+
|
|
133
|
+
| Value | What it finds |
|
|
134
|
+
|---|---|
|
|
135
|
+
| 0–4 | Byte-identical copies, re-encodes, metadata-only edits |
|
|
136
|
+
| **10 (default)** | Resized, recompressed, brightness-shifted, grayscale versions |
|
|
137
|
+
| 14–18 | The same asset cropped or slightly recomposed |
|
|
138
|
+
| >20 | Starts merging genuinely different photos |
|
|
139
|
+
|
|
140
|
+
Screenshots and digital art usually want 6–8. Phone photo libraries need 10–14
|
|
141
|
+
because of sensor noise and HDR processing. **Calibrate on a 200-image sample
|
|
142
|
+
first** — the right value varies by roughly 2x between sources.
|
|
143
|
+
|
|
144
|
+
## How it works
|
|
145
|
+
|
|
146
|
+
Four steps per image, all numpy, no OpenCV:
|
|
147
|
+
|
|
148
|
+
1. **Resize to 32×32 grayscale.** Kills resolution and compression differences.
|
|
149
|
+
2. **2D DCT, keep the top-left 8×8.** Low frequencies describe layout, not pixels.
|
|
150
|
+
3. **Compare each coefficient to the block median.** Above → 1, below → 0. That's 64 bits.
|
|
151
|
+
4. **Distance = popcount of the XOR.** Two fingerprints, one XOR, count the 1s.
|
|
152
|
+
|
|
153
|
+
Two hashes are computed and their distances averaged: `pHash` (DCT, survives resize
|
|
154
|
+
and recompression) and `dHash` (horizontal gradients, sensitive to real structural
|
|
155
|
+
edits). Averaging suppresses most false positives that either would produce alone.
|
|
156
|
+
|
|
157
|
+
### The color veto
|
|
158
|
+
|
|
159
|
+
A 192-float signature is taken by averaging an 8×8 RGB grid, then compared in HSL:
|
|
160
|
+
|
|
161
|
+
```
|
|
162
|
+
distance = 0.55·Δhue·minSat + 0.35·Δsat·minSat + 0.90·Δlightness
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
The `minSat` factor is the important part. Grayscale conversion destroys saturation
|
|
166
|
+
but not perceived lightness, so weighting hue and saturation by the *smaller* of the
|
|
167
|
+
two saturations makes those terms vanish for black-and-white images — only lightness
|
|
168
|
+
is compared. "Photo vs its B&W version" passes; "red apple vs green apple" (both
|
|
169
|
+
saturated) is rejected at full strength.
|
|
170
|
+
|
|
171
|
+
Only the most-different 12 of the 64 cells are averaged, so recoloring one object in
|
|
172
|
+
a corner is not diluted away by 52 unchanged cells.
|
|
173
|
+
|
|
174
|
+
Measured separation: same-image variants score **0.002–0.06**, unrelated images score
|
|
175
|
+
**0.39–0.90**. The default cutoff of 0.18 sits in the gap.
|
|
176
|
+
|
|
177
|
+
### The low-information guard
|
|
178
|
+
|
|
179
|
+
A pHash is only meaningful if the image has low-frequency structure. For solid
|
|
180
|
+
colors, line art, and regular stripes, the AC coefficients are ~0, the median
|
|
181
|
+
comparison degenerates, and every such image hashes identically. `imgsim` measures
|
|
182
|
+
the AC/DC energy ratio and, below 0.02, demands near-identical color (0.06) before
|
|
183
|
+
calling two images similar.
|
|
184
|
+
|
|
185
|
+
## Performance
|
|
186
|
+
|
|
187
|
+
Measured on a 4-core Windows laptop, 1000 JPEGs at 400×400:
|
|
188
|
+
|
|
189
|
+
| Stage | Time |
|
|
190
|
+
|---|---|
|
|
191
|
+
| Decode + fingerprint | 1.7 s |
|
|
192
|
+
| Similarity matrix + clustering | 2.3 s |
|
|
193
|
+
| **Total** | **4.0 s** |
|
|
194
|
+
|
|
195
|
+
Two things make this fast: `Image.draft()` lets libjpeg downsample during decoding
|
|
196
|
+
(5–10× on multi-megapixel photos), and Hamming distance runs through
|
|
197
|
+
`numpy.bitwise_count`, a hardware popcount, over uint64 codes.
|
|
198
|
+
|
|
199
|
+
Memory is a fingerprint per image — 8 bytes of hash plus a 192-float color signature
|
|
200
|
+
— so a million images fit comfortably in RAM.
|
|
201
|
+
|
|
202
|
+
Beyond ~50k images the dense `N×N` matrix becomes the bottleneck (50k → 2.5 GB).
|
|
203
|
+
At that point use `index.iter_pairs()` to stream instead, or bucket by hash prefix
|
|
204
|
+
before comparing. The API leaves room for both.
|
|
205
|
+
|
|
206
|
+
## Semantic mode
|
|
207
|
+
|
|
208
|
+
Perceptual hashing answers *"is this the same picture?"*. It is blind to meaning: two
|
|
209
|
+
photos of the same beach from different angles hash completely differently.
|
|
210
|
+
|
|
211
|
+
```bash
|
|
212
|
+
imgsim ./photos --mode embed --threshold 0.15
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
`embed` mode extracts 512-d ResNet18 features and compares cosine distance. It
|
|
216
|
+
recognises the same subject across viewpoints, at the cost of a PyTorch dependency
|
|
217
|
+
and much weaker fine-grained discrimination — it will happily merge two different
|
|
218
|
+
sunsets. Use it when "similar subject" matters more than "same file".
|
|
219
|
+
|
|
220
|
+
## API at a glance
|
|
221
|
+
|
|
222
|
+
| Member | Purpose |
|
|
223
|
+
|---|---|
|
|
224
|
+
| `ImageIndex(mode="hash")` | Build an index |
|
|
225
|
+
| `.add(path)` / `.add_many(paths)` / `.add_dir(dir)` | Index images (parallel decode) |
|
|
226
|
+
| `.compare(threshold, color_threshold, pairs_limit)` | Group into `Result` |
|
|
227
|
+
| `.nearest(path, k)` | Top-k neighbours of any image |
|
|
228
|
+
| `.save(path)` / `ImageIndex.load(path)` | Persist / restore fingerprints |
|
|
229
|
+
| `.iter_pairs(threshold)` | Stream pairs without materialising them |
|
|
230
|
+
| `.failed` | Paths that could not be decoded |
|
|
231
|
+
| `Result.groups` / `.pairs` / `.duplicates` | Results |
|
|
232
|
+
| `render_html(result)` / `save_html(result, path)` | Report |
|
|
233
|
+
|
|
234
|
+
## Development
|
|
235
|
+
|
|
236
|
+
```bash
|
|
237
|
+
git clone https://github.com/zhouchuan315-png/imgsim
|
|
238
|
+
cd imgsim
|
|
239
|
+
pip install -e ".[dev]"
|
|
240
|
+
pytest
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
Tests build their images from a fixed seed — no binary fixtures in the repo.
|
|
244
|
+
|
|
245
|
+
## License
|
|
246
|
+
|
|
247
|
+
MIT. See [LICENSE](LICENSE).
|
|
248
|
+
|
|
249
|
+
中文文档:[README.zh-CN.md](README.zh-CN.md)
|
|
@@ -0,0 +1,213 @@
|
|
|
1
|
+
# imgsim
|
|
2
|
+
|
|
3
|
+
**Find duplicate and near-duplicate images in a folder — and actually get the right answer.**
|
|
4
|
+
|
|
5
|
+
[](https://github.com/zhouchuan315-png/imgsim/actions/workflows/ci.yml)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
[](pyproject.toml)
|
|
8
|
+
[](https://pypi.org/project/imgsim-py/)
|
|
9
|
+
|
|
10
|
+
`imgsim` indexes a directory of images, computes a 64-bit perceptual fingerprint for
|
|
11
|
+
each one, and groups the ones that are the same picture — resized, recompressed,
|
|
12
|
+
cropped, brightened, or converted to black and white.
|
|
13
|
+
|
|
14
|
+
It exists because plain perceptual hashing has two failure modes that show up in
|
|
15
|
+
every real photo library:
|
|
16
|
+
|
|
17
|
+
| Failure | What you see without a fix |
|
|
18
|
+
|---|---|
|
|
19
|
+
| **Color blindness.** pHash works on grayscale. | A red apple and a green apple of the same shape are reported identical. |
|
|
20
|
+
| **Fingerprint collapse.** Flat images have no low-frequency structure. | A white rectangle and a blue-white stripe pattern hash to the same all-zero fingerprint. |
|
|
21
|
+
|
|
22
|
+
`imgsim` adds a hue-aware color veto and a low-information guard on top of the
|
|
23
|
+
standard pHash + dHash pair. On the bundled benchmark (1000 images, 20 families of
|
|
24
|
+
50 variants each) it recovers **20/20 groups with zero cross-family contamination**.
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## Install
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
pip install imgsim-py
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
The distribution is named `imgsim-py` (the plain `imgsim` name on PyPI belongs to an
|
|
35
|
+
unrelated, unmaintained project), but the import name is `imgsim`:
|
|
36
|
+
|
|
37
|
+
```python
|
|
38
|
+
import imgsim
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Optional semantic mode:
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
pip install imgsim-py[embed] # pulls PyTorch + torchvision (~2 GB)
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Quick start
|
|
48
|
+
|
|
49
|
+
### Command line
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
imgsim ./photos # group duplicates, write ./imgsim-out/report.html
|
|
53
|
+
imgsim ./photos -r -t 8 # recurse, stricter threshold
|
|
54
|
+
imgsim ./photos --query cat.jpg -k 5 # what else looks like cat.jpg?
|
|
55
|
+
imgsim ./photos -f json | jq '.groups | length'
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
`imgsim --help` for everything.
|
|
59
|
+
|
|
60
|
+
### Python
|
|
61
|
+
|
|
62
|
+
```python
|
|
63
|
+
import imgsim
|
|
64
|
+
|
|
65
|
+
index = imgsim.ImageIndex()
|
|
66
|
+
index.add_dir("photos/", recursive=True) # decodes once, in parallel
|
|
67
|
+
|
|
68
|
+
result = index.compare(threshold=10) # Hamming bits, out of 64
|
|
69
|
+
for group in result.groups:
|
|
70
|
+
print(len(group), "similar images")
|
|
71
|
+
for path, distance in zip(group.paths, group.distances):
|
|
72
|
+
print(f" {distance:>6.1f} {path}")
|
|
73
|
+
|
|
74
|
+
imgsim.save_html(result, "report.html")
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Fingerprints are cheap to keep around, so threshold tuning is instant:
|
|
78
|
+
|
|
79
|
+
```python
|
|
80
|
+
index.save("photos.imgsim") # ~300 bytes per image
|
|
81
|
+
index = imgsim.ImageIndex.load("photos.imgsim")
|
|
82
|
+
result = index.compare(threshold=6) # no re-decoding
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Nearest-neighbour lookup for an image that is not in the index:
|
|
86
|
+
|
|
87
|
+
```python
|
|
88
|
+
for path, distance in index.nearest("new-photo.jpg", k=5):
|
|
89
|
+
print(distance, path)
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## Choosing a threshold
|
|
93
|
+
|
|
94
|
+
`threshold` is the maximum Hamming distance between two 64-bit fingerprints —
|
|
95
|
+
literally *how many of the 64 bits are allowed to differ*.
|
|
96
|
+
|
|
97
|
+
| Value | What it finds |
|
|
98
|
+
|---|---|
|
|
99
|
+
| 0–4 | Byte-identical copies, re-encodes, metadata-only edits |
|
|
100
|
+
| **10 (default)** | Resized, recompressed, brightness-shifted, grayscale versions |
|
|
101
|
+
| 14–18 | The same asset cropped or slightly recomposed |
|
|
102
|
+
| >20 | Starts merging genuinely different photos |
|
|
103
|
+
|
|
104
|
+
Screenshots and digital art usually want 6–8. Phone photo libraries need 10–14
|
|
105
|
+
because of sensor noise and HDR processing. **Calibrate on a 200-image sample
|
|
106
|
+
first** — the right value varies by roughly 2x between sources.
|
|
107
|
+
|
|
108
|
+
## How it works
|
|
109
|
+
|
|
110
|
+
Four steps per image, all numpy, no OpenCV:
|
|
111
|
+
|
|
112
|
+
1. **Resize to 32×32 grayscale.** Kills resolution and compression differences.
|
|
113
|
+
2. **2D DCT, keep the top-left 8×8.** Low frequencies describe layout, not pixels.
|
|
114
|
+
3. **Compare each coefficient to the block median.** Above → 1, below → 0. That's 64 bits.
|
|
115
|
+
4. **Distance = popcount of the XOR.** Two fingerprints, one XOR, count the 1s.
|
|
116
|
+
|
|
117
|
+
Two hashes are computed and their distances averaged: `pHash` (DCT, survives resize
|
|
118
|
+
and recompression) and `dHash` (horizontal gradients, sensitive to real structural
|
|
119
|
+
edits). Averaging suppresses most false positives that either would produce alone.
|
|
120
|
+
|
|
121
|
+
### The color veto
|
|
122
|
+
|
|
123
|
+
A 192-float signature is taken by averaging an 8×8 RGB grid, then compared in HSL:
|
|
124
|
+
|
|
125
|
+
```
|
|
126
|
+
distance = 0.55·Δhue·minSat + 0.35·Δsat·minSat + 0.90·Δlightness
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
The `minSat` factor is the important part. Grayscale conversion destroys saturation
|
|
130
|
+
but not perceived lightness, so weighting hue and saturation by the *smaller* of the
|
|
131
|
+
two saturations makes those terms vanish for black-and-white images — only lightness
|
|
132
|
+
is compared. "Photo vs its B&W version" passes; "red apple vs green apple" (both
|
|
133
|
+
saturated) is rejected at full strength.
|
|
134
|
+
|
|
135
|
+
Only the most-different 12 of the 64 cells are averaged, so recoloring one object in
|
|
136
|
+
a corner is not diluted away by 52 unchanged cells.
|
|
137
|
+
|
|
138
|
+
Measured separation: same-image variants score **0.002–0.06**, unrelated images score
|
|
139
|
+
**0.39–0.90**. The default cutoff of 0.18 sits in the gap.
|
|
140
|
+
|
|
141
|
+
### The low-information guard
|
|
142
|
+
|
|
143
|
+
A pHash is only meaningful if the image has low-frequency structure. For solid
|
|
144
|
+
colors, line art, and regular stripes, the AC coefficients are ~0, the median
|
|
145
|
+
comparison degenerates, and every such image hashes identically. `imgsim` measures
|
|
146
|
+
the AC/DC energy ratio and, below 0.02, demands near-identical color (0.06) before
|
|
147
|
+
calling two images similar.
|
|
148
|
+
|
|
149
|
+
## Performance
|
|
150
|
+
|
|
151
|
+
Measured on a 4-core Windows laptop, 1000 JPEGs at 400×400:
|
|
152
|
+
|
|
153
|
+
| Stage | Time |
|
|
154
|
+
|---|---|
|
|
155
|
+
| Decode + fingerprint | 1.7 s |
|
|
156
|
+
| Similarity matrix + clustering | 2.3 s |
|
|
157
|
+
| **Total** | **4.0 s** |
|
|
158
|
+
|
|
159
|
+
Two things make this fast: `Image.draft()` lets libjpeg downsample during decoding
|
|
160
|
+
(5–10× on multi-megapixel photos), and Hamming distance runs through
|
|
161
|
+
`numpy.bitwise_count`, a hardware popcount, over uint64 codes.
|
|
162
|
+
|
|
163
|
+
Memory is a fingerprint per image — 8 bytes of hash plus a 192-float color signature
|
|
164
|
+
— so a million images fit comfortably in RAM.
|
|
165
|
+
|
|
166
|
+
Beyond ~50k images the dense `N×N` matrix becomes the bottleneck (50k → 2.5 GB).
|
|
167
|
+
At that point use `index.iter_pairs()` to stream instead, or bucket by hash prefix
|
|
168
|
+
before comparing. The API leaves room for both.
|
|
169
|
+
|
|
170
|
+
## Semantic mode
|
|
171
|
+
|
|
172
|
+
Perceptual hashing answers *"is this the same picture?"*. It is blind to meaning: two
|
|
173
|
+
photos of the same beach from different angles hash completely differently.
|
|
174
|
+
|
|
175
|
+
```bash
|
|
176
|
+
imgsim ./photos --mode embed --threshold 0.15
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
`embed` mode extracts 512-d ResNet18 features and compares cosine distance. It
|
|
180
|
+
recognises the same subject across viewpoints, at the cost of a PyTorch dependency
|
|
181
|
+
and much weaker fine-grained discrimination — it will happily merge two different
|
|
182
|
+
sunsets. Use it when "similar subject" matters more than "same file".
|
|
183
|
+
|
|
184
|
+
## API at a glance
|
|
185
|
+
|
|
186
|
+
| Member | Purpose |
|
|
187
|
+
|---|---|
|
|
188
|
+
| `ImageIndex(mode="hash")` | Build an index |
|
|
189
|
+
| `.add(path)` / `.add_many(paths)` / `.add_dir(dir)` | Index images (parallel decode) |
|
|
190
|
+
| `.compare(threshold, color_threshold, pairs_limit)` | Group into `Result` |
|
|
191
|
+
| `.nearest(path, k)` | Top-k neighbours of any image |
|
|
192
|
+
| `.save(path)` / `ImageIndex.load(path)` | Persist / restore fingerprints |
|
|
193
|
+
| `.iter_pairs(threshold)` | Stream pairs without materialising them |
|
|
194
|
+
| `.failed` | Paths that could not be decoded |
|
|
195
|
+
| `Result.groups` / `.pairs` / `.duplicates` | Results |
|
|
196
|
+
| `render_html(result)` / `save_html(result, path)` | Report |
|
|
197
|
+
|
|
198
|
+
## Development
|
|
199
|
+
|
|
200
|
+
```bash
|
|
201
|
+
git clone https://github.com/zhouchuan315-png/imgsim
|
|
202
|
+
cd imgsim
|
|
203
|
+
pip install -e ".[dev]"
|
|
204
|
+
pytest
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
Tests build their images from a fixed seed — no binary fixtures in the repo.
|
|
208
|
+
|
|
209
|
+
## License
|
|
210
|
+
|
|
211
|
+
MIT. See [LICENSE](LICENSE).
|
|
212
|
+
|
|
213
|
+
中文文档:[README.zh-CN.md](README.zh-CN.md)
|