lensgrep 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of lensgrep might be problematic. Click here for more details.
- lensgrep-0.1.0/.github/workflows/ci.yml +32 -0
- lensgrep-0.1.0/.gitignore +31 -0
- lensgrep-0.1.0/LICENSE +21 -0
- lensgrep-0.1.0/PKG-INFO +307 -0
- lensgrep-0.1.0/README.md +257 -0
- lensgrep-0.1.0/pyproject.toml +105 -0
- lensgrep-0.1.0/src/lensgrep/__init__.py +3 -0
- lensgrep-0.1.0/src/lensgrep/__main__.py +6 -0
- lensgrep-0.1.0/src/lensgrep/cli.py +344 -0
- lensgrep-0.1.0/src/lensgrep/core/__init__.py +1 -0
- lensgrep-0.1.0/src/lensgrep/core/cluster.py +44 -0
- lensgrep-0.1.0/src/lensgrep/core/db.py +468 -0
- lensgrep-0.1.0/src/lensgrep/core/dedupe.py +370 -0
- lensgrep-0.1.0/src/lensgrep/core/embedder.py +101 -0
- lensgrep-0.1.0/src/lensgrep/core/paths.py +29 -0
- lensgrep-0.1.0/src/lensgrep/core/scanner.py +254 -0
- lensgrep-0.1.0/src/lensgrep/core/search.py +65 -0
- lensgrep-0.1.0/src/lensgrep/exporters/__init__.py +1 -0
- lensgrep-0.1.0/src/lensgrep/exporters/crop.py +52 -0
- lensgrep-0.1.0/src/lensgrep/exporters/files.py +54 -0
- lensgrep-0.1.0/src/lensgrep/exporters/lora.py +253 -0
- lensgrep-0.1.0/tests/__init__.py +0 -0
- lensgrep-0.1.0/tests/test_crop.py +25 -0
- lensgrep-0.1.0/tests/test_db.py +95 -0
- lensgrep-0.1.0/tests/test_dedupe.py +190 -0
- lensgrep-0.1.0/tests/test_safety.py +409 -0
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
pull_request:
|
|
7
|
+
branches: [main]
|
|
8
|
+
|
|
9
|
+
jobs:
|
|
10
|
+
lint-test:
|
|
11
|
+
runs-on: ${{ matrix.os }}
|
|
12
|
+
strategy:
|
|
13
|
+
fail-fast: false
|
|
14
|
+
matrix:
|
|
15
|
+
os: [ubuntu-latest, windows-latest, macos-latest]
|
|
16
|
+
python-version: ["3.10", "3.11", "3.12"]
|
|
17
|
+
steps:
|
|
18
|
+
- uses: actions/checkout@v4
|
|
19
|
+
- uses: actions/setup-python@v5
|
|
20
|
+
with:
|
|
21
|
+
python-version: ${{ matrix.python-version }}
|
|
22
|
+
cache: pip
|
|
23
|
+
- name: Install
|
|
24
|
+
run: |
|
|
25
|
+
python -m pip install --upgrade pip
|
|
26
|
+
pip install -e ".[dev]"
|
|
27
|
+
- name: Lint (ruff)
|
|
28
|
+
run: ruff check .
|
|
29
|
+
- name: Format check (ruff)
|
|
30
|
+
run: ruff format --check .
|
|
31
|
+
- name: Tests
|
|
32
|
+
run: pytest -q
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
__pycache__/
|
|
2
|
+
*.py[cod]
|
|
3
|
+
*$py.class
|
|
4
|
+
*.so
|
|
5
|
+
.Python
|
|
6
|
+
build/
|
|
7
|
+
dist/
|
|
8
|
+
*.egg-info/
|
|
9
|
+
.eggs/
|
|
10
|
+
.venv/
|
|
11
|
+
venv/
|
|
12
|
+
.env
|
|
13
|
+
.coverage
|
|
14
|
+
htmlcov/
|
|
15
|
+
.pytest_cache/
|
|
16
|
+
.ruff_cache/
|
|
17
|
+
.mypy_cache/
|
|
18
|
+
.idea/
|
|
19
|
+
.vscode/
|
|
20
|
+
*.swp
|
|
21
|
+
*.swo
|
|
22
|
+
.DS_Store
|
|
23
|
+
Thumbs.db
|
|
24
|
+
# Local data
|
|
25
|
+
*.db
|
|
26
|
+
*.sqlite
|
|
27
|
+
*.sqlite3
|
|
28
|
+
*.faiss
|
|
29
|
+
.lensgrep/
|
|
30
|
+
# Models (download on first run)
|
|
31
|
+
*.onnx
|
lensgrep-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Krzysiek Ołdak
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
lensgrep-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,307 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: lensgrep
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Grep for your photo library — find any person across thousands of photos, locally and privately. Built-in LoRA dataset builder.
|
|
5
|
+
Project-URL: Homepage, https://github.com/Kris-O/lensgrep
|
|
6
|
+
Project-URL: Repository, https://github.com/Kris-O/lensgrep
|
|
7
|
+
Project-URL: Issues, https://github.com/Kris-O/lensgrep/issues
|
|
8
|
+
Project-URL: Changelog, https://github.com/Kris-O/lensgrep/blob/main/CHANGELOG.md
|
|
9
|
+
Author-email: Krzysiek Ołdak <krzysiek.oldak@gmail.com>
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: arcface,cli,face-recognition,flux,insightface,lora,photo-library,privacy,stable-diffusion
|
|
13
|
+
Classifier: Development Status :: 3 - Alpha
|
|
14
|
+
Classifier: Environment :: Console
|
|
15
|
+
Classifier: Intended Audience :: Developers
|
|
16
|
+
Classifier: Intended Audience :: End Users/Desktop
|
|
17
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
18
|
+
Classifier: Operating System :: OS Independent
|
|
19
|
+
Classifier: Programming Language :: Python :: 3
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
23
|
+
Classifier: Topic :: Multimedia :: Graphics
|
|
24
|
+
Classifier: Topic :: Scientific/Engineering :: Image Recognition
|
|
25
|
+
Classifier: Topic :: Utilities
|
|
26
|
+
Requires-Python: >=3.10
|
|
27
|
+
Requires-Dist: faiss-cpu>=1.8.0
|
|
28
|
+
Requires-Dist: imagehash>=4.3.1
|
|
29
|
+
Requires-Dist: insightface>=0.7.3
|
|
30
|
+
Requires-Dist: numpy>=1.26.0
|
|
31
|
+
Requires-Dist: onnxruntime>=1.17.0
|
|
32
|
+
Requires-Dist: opencv-python>=4.9.0
|
|
33
|
+
Requires-Dist: pillow>=10.0.0
|
|
34
|
+
Requires-Dist: platformdirs>=4.2.0
|
|
35
|
+
Requires-Dist: rich>=13.7.0
|
|
36
|
+
Requires-Dist: scikit-learn>=1.4.0
|
|
37
|
+
Requires-Dist: tqdm>=4.66.0
|
|
38
|
+
Requires-Dist: typer>=0.12.0
|
|
39
|
+
Provides-Extra: caption
|
|
40
|
+
Requires-Dist: torch>=2.2.0; extra == 'caption'
|
|
41
|
+
Requires-Dist: transformers>=4.40.0; extra == 'caption'
|
|
42
|
+
Provides-Extra: dev
|
|
43
|
+
Requires-Dist: mypy>=1.10.0; extra == 'dev'
|
|
44
|
+
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
|
|
45
|
+
Requires-Dist: pytest>=8.0.0; extra == 'dev'
|
|
46
|
+
Requires-Dist: ruff>=0.5.0; extra == 'dev'
|
|
47
|
+
Provides-Extra: gpu
|
|
48
|
+
Requires-Dist: onnxruntime-gpu>=1.17.0; extra == 'gpu'
|
|
49
|
+
Description-Content-Type: text/markdown
|
|
50
|
+
|
|
51
|
+
# lensgrep
|
|
52
|
+
|
|
53
|
+
> **`grep` for your photo library.** Find people. Remove duplicates. Build LoRA datasets. All locally, all private, no cloud.
|
|
54
|
+
|
|
55
|
+
[](https://pypi.org/project/lensgrep/)
|
|
56
|
+
[](https://pypi.org/project/lensgrep/)
|
|
57
|
+
[](LICENSE)
|
|
58
|
+
[](https://github.com/Kris-O/lensgrep/actions/workflows/ci.yml)
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
$ lensgrep scan ~/Photos
|
|
62
|
+
scanned 12,847 files faces+= 38,219 clustered 412 groups
|
|
63
|
+
|
|
64
|
+
$ lensgrep name 3 "Mom"
|
|
65
|
+
✓ 'Mom' ← cluster 3, assigned 1,204 face(s)
|
|
66
|
+
|
|
67
|
+
$ lensgrep find Mom -n 5 --paths
|
|
68
|
+
/home/me/Photos/2023/birthday/IMG_0341.jpg
|
|
69
|
+
/home/me/Photos/2024/cabin/DSC_8821.jpg
|
|
70
|
+
/home/me/Photos/2024/cabin/DSC_8824.jpg
|
|
71
|
+
/home/me/Photos/2025/holidays/IMG_1129.jpg
|
|
72
|
+
/home/me/Photos/2025/holidays/IMG_1131.jpg
|
|
73
|
+
|
|
74
|
+
$ lensgrep lora Mom ./mom_dataset --size 50
|
|
75
|
+
✓ LoRA dataset: 50 images at 1024×1024 → ./mom_dataset
|
|
76
|
+
Token: mom
|
|
77
|
+
Next: open AI Toolkit / kohya_ss → point dataset path here → train.
|
|
78
|
+
|
|
79
|
+
$ lensgrep dupes --near --threshold 5
|
|
80
|
+
Found 412 group(s), 1,847 extra file(s), could free 14,203.5 MB.
|
|
81
|
+
…
|
|
82
|
+
Read-only listing. Re-run with --mode delete or --mode symlink to act.
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
## What it is
|
|
86
|
+
|
|
87
|
+
**lensgrep** is a single-purpose tool that does one thing well: **find photos of a specific person in a folder of photos**, no matter how big the folder is. SQLite under the hood, [InsightFace](https://github.com/deepinsight/insightface) ArcFace embeddings, FAISS for vector search. Nothing leaves your machine.
|
|
88
|
+
|
|
89
|
+
It's intentionally small. If you want a full self-hosted Google Photos clone, run [PhotoPrism](https://github.com/photoprism/photoprism) or [Immich](https://github.com/immich-app/immich). If you want `grep` for faces, run this.
|
|
90
|
+
|
|
91
|
+
## Killer features
|
|
92
|
+
|
|
93
|
+
### Duplicate detection — reclaim disk space
|
|
94
|
+
|
|
95
|
+
Most photo libraries have 10–30% wasted space: phone-sync dupes, edited copies, burst shots, JPEG-vs-RAW pairs, "the same file in three folders because cloud sync went sideways."
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
lensgrep dupes # exact (bit-identical) only — read-only listing
|
|
99
|
+
lensgrep dupes --near --threshold 5 # near-duplicates: resized / re-saved / minor edits
|
|
100
|
+
lensgrep dupes --near --mode delete # actually delete the victims
|
|
101
|
+
lensgrep dupes --near --mode symlink # replace victims with symlinks to the keeper
|
|
102
|
+
lensgrep dupes --keep newest # alternative: oldest | newest | shortest-path | largest (default)
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
**Two detection modes:**
|
|
106
|
+
- **Exact** — SHA-256 hash of the file, zero false positives, catches bit-identical copies regardless of filename.
|
|
107
|
+
- **Near** — perceptual hash (pHash) with configurable Hamming-distance threshold. Catches resized copies, JPEG re-saves, minor color edits, thumbnails — the stuff that exact match misses.
|
|
108
|
+
|
|
109
|
+
**Safety by default:** without `--mode delete` or `--mode symlink`, the command is read-only. You'll always see what it *would* do before it does anything.
|
|
110
|
+
|
|
111
|
+
### LoRA dataset builder — flagship
|
|
112
|
+
|
|
113
|
+
Training a face LoRA (Flux / SDXL) usually means manually picking 40–60 diverse photos of one person from your library. lensgrep does this in one command:
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
lensgrep lora "Krzysiek" ./dataset --size 50 --resolution 1024
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
This:
|
|
120
|
+
1. Pulls all photos containing Krzysiek from the indexed library
|
|
121
|
+
2. Scores quality (sharpness, face area, detection confidence)
|
|
122
|
+
3. Picks 50 **diverse** photos via greedy farthest-point sampling on ArcFace embeddings — no near-duplicates, balanced mix of closeup / portrait / half-body / full-body
|
|
123
|
+
4. Crops to 1024×1024 with the face in the upper third (rule-of-thirds)
|
|
124
|
+
5. Writes `.txt` caption files next to each image, ready for [AI Toolkit](https://github.com/ostris/ai-toolkit) or [kohya_ss](https://github.com/bmaltais/kohya_ss)
|
|
125
|
+
|
|
126
|
+
You go from "8 000 raw photos" to "trainable Flux LoRA dataset" in **one command**.
|
|
127
|
+
|
|
128
|
+
## Install
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
pip install lensgrep
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Optional: GPU support (10× faster scan)
|
|
135
|
+
|
|
136
|
+
```bash
|
|
137
|
+
pip install "lensgrep[gpu]"
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
Requires Python 3.10+. First run downloads ~280 MB of InsightFace models to `~/.insightface/`.
|
|
141
|
+
|
|
142
|
+
## Quickstart
|
|
143
|
+
|
|
144
|
+
```bash
|
|
145
|
+
# 1. scan your photo library (one time, ~1.5h CPU / ~10 min GPU for 10k photos)
|
|
146
|
+
lensgrep scan ~/Photos
|
|
147
|
+
|
|
148
|
+
# 2. browse detected face clusters
|
|
149
|
+
lensgrep people --min-size 10
|
|
150
|
+
|
|
151
|
+
# 3. name the ones you care about
|
|
152
|
+
lensgrep name 3 "Mom"
|
|
153
|
+
lensgrep name 7 "Dad"
|
|
154
|
+
lensgrep name 12 "Krzysiek"
|
|
155
|
+
|
|
156
|
+
# 4. find photos of someone
|
|
157
|
+
lensgrep find Mom -n 100
|
|
158
|
+
lensgrep find Mom --paths | xargs -I{} cp {} ./mom_photos/
|
|
159
|
+
|
|
160
|
+
# 5. export all photos of a person
|
|
161
|
+
lensgrep export Mom ./mom_photos --mode symlink
|
|
162
|
+
|
|
163
|
+
# 6. build a LoRA training dataset
|
|
164
|
+
lensgrep lora Krzysiek ./krzysiek_dataset --size 50
|
|
165
|
+
|
|
166
|
+
# 7. quick stats
|
|
167
|
+
lensgrep stats
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
## Commands
|
|
171
|
+
|
|
172
|
+
| Command | What it does |
|
|
173
|
+
|---------|--------------|
|
|
174
|
+
| `scan <PATH>` | Recursively scan folders for faces. Incremental by default (skips unchanged files). |
|
|
175
|
+
| `people` | List detected face clusters, with face counts. |
|
|
176
|
+
| `name <CLUSTER> <NAME>` | Tag a cluster with a person's name. |
|
|
177
|
+
| `find <NAME>` | List photos of a tagged person, ranked by face quality. |
|
|
178
|
+
| `export <NAME> <DIR>` | Copy/symlink/hardlink all matching photos to a folder. |
|
|
179
|
+
| `lora <NAME> <DIR>` | Build a Flux/SDXL LoRA training dataset. |
|
|
180
|
+
| `dupes` | Find (and optionally remove) duplicate photos — exact or perceptual. |
|
|
181
|
+
| `recluster` | Re-run DBSCAN with custom parameters. |
|
|
182
|
+
| `stats` | Database statistics. |
|
|
183
|
+
|
|
184
|
+
Run `lensgrep <command> --help` for full options on any command.
|
|
185
|
+
|
|
186
|
+
## How it works
|
|
187
|
+
|
|
188
|
+
```
|
|
189
|
+
┌────────────────┐
|
|
190
|
+
~/Photos ───▶ │ scan │ ──▶ images table (SQLite)
|
|
191
|
+
│ - detect │
|
|
192
|
+
│ - embed (512d)│ ──▶ faces table (SQLite, embedding as BLOB)
|
|
193
|
+
│ - score │
|
|
194
|
+
└────────────────┘
|
|
195
|
+
│
|
|
196
|
+
▼
|
|
197
|
+
┌────────────────┐
|
|
198
|
+
│ cluster │ ──▶ persons table (named clusters)
|
|
199
|
+
│ (DBSCAN) │
|
|
200
|
+
└────────────────┘
|
|
201
|
+
│
|
|
202
|
+
▼
|
|
203
|
+
┌────────────────┐
|
|
204
|
+
│ find / export │
|
|
205
|
+
│ / lora │
|
|
206
|
+
└────────────────┘
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
- **Detection + embedding:** InsightFace `buffalo_l` (SCRFD detector + ArcFace r50 recognition). State-of-the-art, ONNX, runs on CPU or CUDA.
|
|
210
|
+
- **Storage:** Plain SQLite — no Docker, no daemons, no cloud accounts. The whole index lives in one file at the platform-standard data dir.
|
|
211
|
+
- **Vector search:** FAISS `IndexFlatIP` on L2-normalized embeddings — exact cosine similarity, sub-millisecond on a million faces.
|
|
212
|
+
- **Clustering:** DBSCAN over cosine distance, `eps=0.35`, `min_samples=5`. Tune via `lensgrep recluster --eps 0.4 --min-samples 8`.
|
|
213
|
+
- **LoRA selection:** Greedy farthest-point sampling on embeddings — maximizes diversity. Quality scoring weights sharpness (Laplacian variance), face area, detection confidence.
|
|
214
|
+
|
|
215
|
+
## Why this exists
|
|
216
|
+
|
|
217
|
+
In 2026, the photo-management landscape has two extremes:
|
|
218
|
+
|
|
219
|
+
- **Bloated cloud apps** that ingest your library and "do AI on it": Google Photos, Apple Photos, Amazon Photos. Hand over your private life.
|
|
220
|
+
- **Self-hosted galleries with face features bolted on**: PhotoPrism, Immich, digiKam. Powerful but heavy — Docker, databases, web UIs, hours of setup, weeks of curating.
|
|
221
|
+
|
|
222
|
+
Neither helps if you just want to answer one question:
|
|
223
|
+
|
|
224
|
+
> "Where are all the photos of [person] in this big folder?"
|
|
225
|
+
|
|
226
|
+
…or:
|
|
227
|
+
|
|
228
|
+
> "Why does this folder weigh 80 GB? How many of these photos are actual duplicates?"
|
|
229
|
+
|
|
230
|
+
…or:
|
|
231
|
+
|
|
232
|
+
> "I'm training a Flux LoRA of my partner. Give me 50 diverse, high-quality photos."
|
|
233
|
+
|
|
234
|
+
That's the niche. **One job. Done well.**
|
|
235
|
+
|
|
236
|
+
## Comparison
|
|
237
|
+
|
|
238
|
+
| | lensgrep | PhotoPrism | Immich | digiKam | face_recognition |
|
|
239
|
+
|---|---|---|---|---|---|
|
|
240
|
+
| CLI-first | ✅ | ❌ | ❌ | ❌ | ✅ (library) |
|
|
241
|
+
| No Docker / DB setup | ✅ | ❌ | ❌ | ✅ | ✅ |
|
|
242
|
+
| Duplicate detection (exact + perceptual) | ✅ | partial | partial | partial | ❌ |
|
|
243
|
+
| LoRA dataset export | ✅ | ❌ | ❌ | ❌ | ❌ |
|
|
244
|
+
| FAISS vector search | ✅ | ❌ | ✅ | ❌ | ❌ |
|
|
245
|
+
| Modern face model (ArcFace) | ✅ | ✅ | ✅ | ❌ (older) | ❌ (dlib) |
|
|
246
|
+
| Local-only / no cloud | ✅ | ✅ | ✅ | ✅ | ✅ |
|
|
247
|
+
| Install in 10 seconds | ✅ | ❌ | ❌ | ❌ | ✅ |
|
|
248
|
+
|
|
249
|
+
## Privacy
|
|
250
|
+
|
|
251
|
+
Everything stays on your machine. No telemetry, no analytics, no network calls except the one-time model download from GitHub releases on first run. The SQLite database is plain text — you can `sqlite3 lensgrep.db ".dump"` and read it.
|
|
252
|
+
|
|
253
|
+
If you want to verify: `pip install lensgrep` ships ~600 lines of pure Python source. Audit it in an afternoon.
|
|
254
|
+
|
|
255
|
+
## Performance
|
|
256
|
+
|
|
257
|
+
Rough numbers on a 2020 laptop (Quadro RTX 5000 mobile, 16 GB VRAM):
|
|
258
|
+
|
|
259
|
+
| Library size | CPU scan | GPU scan (CUDA) |
|
|
260
|
+
|---|---|---|
|
|
261
|
+
| 1 000 photos | ~3 min | ~30 s |
|
|
262
|
+
| 10 000 photos | ~30 min | ~5 min |
|
|
263
|
+
| 100 000 photos | ~5 h | ~50 min |
|
|
264
|
+
|
|
265
|
+
Scan is the only slow part. After indexing, `find` / `export` / `lora` are sub-second on hundreds of thousands of faces.
|
|
266
|
+
|
|
267
|
+
CPU works fine if you're willing to leave it running overnight once. GPU setup needs CUDA Toolkit 12 + cuDNN 9 (`pip install "lensgrep[gpu]"` is necessary but not sufficient — see [docs/gpu.md](docs/gpu.md)).
|
|
268
|
+
|
|
269
|
+
## Roadmap
|
|
270
|
+
|
|
271
|
+
- [x] CLI MVP (scan, people, name, find, export, lora, dupes, stats)
|
|
272
|
+
- [ ] Per-person nearest-neighbor add: `lensgrep find-similar <photo.jpg>`
|
|
273
|
+
- [ ] EXIF date timeline filter: `lensgrep find Mom --year 2024`
|
|
274
|
+
- [ ] EXIF GPS map output
|
|
275
|
+
- [ ] HEIC support
|
|
276
|
+
- [ ] `lensgrep serve` — read-only web UI (FastAPI + minimal frontend) for browsing
|
|
277
|
+
- [ ] BLIP / Florence-2 auto-captions for richer LoRA training data
|
|
278
|
+
|
|
279
|
+
If the project gains traction, the web UI is next. Open an issue if you want to push it forward.
|
|
280
|
+
|
|
281
|
+
## Contributing
|
|
282
|
+
|
|
283
|
+
PRs welcome. Project conventions:
|
|
284
|
+
|
|
285
|
+
- Ruff for lint + format (`ruff check . && ruff format .`)
|
|
286
|
+
- Pytest for tests (`pytest`)
|
|
287
|
+
- Conventional commits (`feat:`, `fix:`, `docs:`, `refactor:`)
|
|
288
|
+
|
|
289
|
+
```bash
|
|
290
|
+
git clone https://github.com/Kris-O/lensgrep
|
|
291
|
+
cd lensgrep
|
|
292
|
+
pip install -e ".[dev]"
|
|
293
|
+
pytest
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
## License
|
|
297
|
+
|
|
298
|
+
MIT — see [LICENSE](LICENSE).
|
|
299
|
+
|
|
300
|
+
## Acknowledgments
|
|
301
|
+
|
|
302
|
+
Standing on the shoulders of:
|
|
303
|
+
- [InsightFace](https://github.com/deepinsight/insightface) — face detection + recognition
|
|
304
|
+
- [FAISS](https://github.com/facebookresearch/faiss) — vector search
|
|
305
|
+
- [Typer](https://github.com/fastapi/typer) + [Rich](https://github.com/Textualize/rich) — joy of CLI dev
|
|
306
|
+
|
|
307
|
+
Inspired by the question: *why isn't this just one command yet?*
|
lensgrep-0.1.0/README.md
ADDED
|
@@ -0,0 +1,257 @@
|
|
|
1
|
+
# lensgrep
|
|
2
|
+
|
|
3
|
+
> **`grep` for your photo library.** Find people. Remove duplicates. Build LoRA datasets. All locally, all private, no cloud.
|
|
4
|
+
|
|
5
|
+
[](https://pypi.org/project/lensgrep/)
|
|
6
|
+
[](https://pypi.org/project/lensgrep/)
|
|
7
|
+
[](LICENSE)
|
|
8
|
+
[](https://github.com/Kris-O/lensgrep/actions/workflows/ci.yml)
|
|
9
|
+
|
|
10
|
+
```
|
|
11
|
+
$ lensgrep scan ~/Photos
|
|
12
|
+
scanned 12,847 files faces+= 38,219 clustered 412 groups
|
|
13
|
+
|
|
14
|
+
$ lensgrep name 3 "Mom"
|
|
15
|
+
✓ 'Mom' ← cluster 3, assigned 1,204 face(s)
|
|
16
|
+
|
|
17
|
+
$ lensgrep find Mom -n 5 --paths
|
|
18
|
+
/home/me/Photos/2023/birthday/IMG_0341.jpg
|
|
19
|
+
/home/me/Photos/2024/cabin/DSC_8821.jpg
|
|
20
|
+
/home/me/Photos/2024/cabin/DSC_8824.jpg
|
|
21
|
+
/home/me/Photos/2025/holidays/IMG_1129.jpg
|
|
22
|
+
/home/me/Photos/2025/holidays/IMG_1131.jpg
|
|
23
|
+
|
|
24
|
+
$ lensgrep lora Mom ./mom_dataset --size 50
|
|
25
|
+
✓ LoRA dataset: 50 images at 1024×1024 → ./mom_dataset
|
|
26
|
+
Token: mom
|
|
27
|
+
Next: open AI Toolkit / kohya_ss → point dataset path here → train.
|
|
28
|
+
|
|
29
|
+
$ lensgrep dupes --near --threshold 5
|
|
30
|
+
Found 412 group(s), 1,847 extra file(s), could free 14,203.5 MB.
|
|
31
|
+
…
|
|
32
|
+
Read-only listing. Re-run with --mode delete or --mode symlink to act.
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
## What it is
|
|
36
|
+
|
|
37
|
+
**lensgrep** is a single-purpose tool that does one thing well: **find photos of a specific person in a folder of photos**, no matter how big the folder is. SQLite under the hood, [InsightFace](https://github.com/deepinsight/insightface) ArcFace embeddings, FAISS for vector search. Nothing leaves your machine.
|
|
38
|
+
|
|
39
|
+
It's intentionally small. If you want a full self-hosted Google Photos clone, run [PhotoPrism](https://github.com/photoprism/photoprism) or [Immich](https://github.com/immich-app/immich). If you want `grep` for faces, run this.
|
|
40
|
+
|
|
41
|
+
## Killer features
|
|
42
|
+
|
|
43
|
+
### Duplicate detection — reclaim disk space
|
|
44
|
+
|
|
45
|
+
Most photo libraries have 10–30% wasted space: phone-sync dupes, edited copies, burst shots, JPEG-vs-RAW pairs, "the same file in three folders because cloud sync went sideways."
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
lensgrep dupes # exact (bit-identical) only — read-only listing
|
|
49
|
+
lensgrep dupes --near --threshold 5 # near-duplicates: resized / re-saved / minor edits
|
|
50
|
+
lensgrep dupes --near --mode delete # actually delete the victims
|
|
51
|
+
lensgrep dupes --near --mode symlink # replace victims with symlinks to the keeper
|
|
52
|
+
lensgrep dupes --keep newest # alternative: oldest | newest | shortest-path | largest (default)
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
**Two detection modes:**
|
|
56
|
+
- **Exact** — SHA-256 hash of the file, zero false positives, catches bit-identical copies regardless of filename.
|
|
57
|
+
- **Near** — perceptual hash (pHash) with configurable Hamming-distance threshold. Catches resized copies, JPEG re-saves, minor color edits, thumbnails — the stuff that exact match misses.
|
|
58
|
+
|
|
59
|
+
**Safety by default:** without `--mode delete` or `--mode symlink`, the command is read-only. You'll always see what it *would* do before it does anything.
|
|
60
|
+
|
|
61
|
+
### LoRA dataset builder — flagship
|
|
62
|
+
|
|
63
|
+
Training a face LoRA (Flux / SDXL) usually means manually picking 40–60 diverse photos of one person from your library. lensgrep does this in one command:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
lensgrep lora "Krzysiek" ./dataset --size 50 --resolution 1024
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
This:
|
|
70
|
+
1. Pulls all photos containing Krzysiek from the indexed library
|
|
71
|
+
2. Scores quality (sharpness, face area, detection confidence)
|
|
72
|
+
3. Picks 50 **diverse** photos via greedy farthest-point sampling on ArcFace embeddings — no near-duplicates, balanced mix of closeup / portrait / half-body / full-body
|
|
73
|
+
4. Crops to 1024×1024 with the face in the upper third (rule-of-thirds)
|
|
74
|
+
5. Writes `.txt` caption files next to each image, ready for [AI Toolkit](https://github.com/ostris/ai-toolkit) or [kohya_ss](https://github.com/bmaltais/kohya_ss)
|
|
75
|
+
|
|
76
|
+
You go from "8 000 raw photos" to "trainable Flux LoRA dataset" in **one command**.
|
|
77
|
+
|
|
78
|
+
## Install
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
pip install lensgrep
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Optional: GPU support (10× faster scan)
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
pip install "lensgrep[gpu]"
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Requires Python 3.10+. First run downloads ~280 MB of InsightFace models to `~/.insightface/`.
|
|
91
|
+
|
|
92
|
+
## Quickstart
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
# 1. scan your photo library (one time, ~1.5h CPU / ~10 min GPU for 10k photos)
|
|
96
|
+
lensgrep scan ~/Photos
|
|
97
|
+
|
|
98
|
+
# 2. browse detected face clusters
|
|
99
|
+
lensgrep people --min-size 10
|
|
100
|
+
|
|
101
|
+
# 3. name the ones you care about
|
|
102
|
+
lensgrep name 3 "Mom"
|
|
103
|
+
lensgrep name 7 "Dad"
|
|
104
|
+
lensgrep name 12 "Krzysiek"
|
|
105
|
+
|
|
106
|
+
# 4. find photos of someone
|
|
107
|
+
lensgrep find Mom -n 100
|
|
108
|
+
lensgrep find Mom --paths | xargs -I{} cp {} ./mom_photos/
|
|
109
|
+
|
|
110
|
+
# 5. export all photos of a person
|
|
111
|
+
lensgrep export Mom ./mom_photos --mode symlink
|
|
112
|
+
|
|
113
|
+
# 6. build a LoRA training dataset
|
|
114
|
+
lensgrep lora Krzysiek ./krzysiek_dataset --size 50
|
|
115
|
+
|
|
116
|
+
# 7. quick stats
|
|
117
|
+
lensgrep stats
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
## Commands
|
|
121
|
+
|
|
122
|
+
| Command | What it does |
|
|
123
|
+
|---------|--------------|
|
|
124
|
+
| `scan <PATH>` | Recursively scan folders for faces. Incremental by default (skips unchanged files). |
|
|
125
|
+
| `people` | List detected face clusters, with face counts. |
|
|
126
|
+
| `name <CLUSTER> <NAME>` | Tag a cluster with a person's name. |
|
|
127
|
+
| `find <NAME>` | List photos of a tagged person, ranked by face quality. |
|
|
128
|
+
| `export <NAME> <DIR>` | Copy/symlink/hardlink all matching photos to a folder. |
|
|
129
|
+
| `lora <NAME> <DIR>` | Build a Flux/SDXL LoRA training dataset. |
|
|
130
|
+
| `dupes` | Find (and optionally remove) duplicate photos — exact or perceptual. |
|
|
131
|
+
| `recluster` | Re-run DBSCAN with custom parameters. |
|
|
132
|
+
| `stats` | Database statistics. |
|
|
133
|
+
|
|
134
|
+
Run `lensgrep <command> --help` for full options on any command.
|
|
135
|
+
|
|
136
|
+
## How it works
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
┌────────────────┐
|
|
140
|
+
~/Photos ───▶ │ scan │ ──▶ images table (SQLite)
|
|
141
|
+
│ - detect │
|
|
142
|
+
│ - embed (512d)│ ──▶ faces table (SQLite, embedding as BLOB)
|
|
143
|
+
│ - score │
|
|
144
|
+
└────────────────┘
|
|
145
|
+
│
|
|
146
|
+
▼
|
|
147
|
+
┌────────────────┐
|
|
148
|
+
│ cluster │ ──▶ persons table (named clusters)
|
|
149
|
+
│ (DBSCAN) │
|
|
150
|
+
└────────────────┘
|
|
151
|
+
│
|
|
152
|
+
▼
|
|
153
|
+
┌────────────────┐
|
|
154
|
+
│ find / export │
|
|
155
|
+
│ / lora │
|
|
156
|
+
└────────────────┘
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
- **Detection + embedding:** InsightFace `buffalo_l` (SCRFD detector + ArcFace r50 recognition). State-of-the-art, ONNX, runs on CPU or CUDA.
|
|
160
|
+
- **Storage:** Plain SQLite — no Docker, no daemons, no cloud accounts. The whole index lives in one file at the platform-standard data dir.
|
|
161
|
+
- **Vector search:** FAISS `IndexFlatIP` on L2-normalized embeddings — exact cosine similarity, sub-millisecond on a million faces.
|
|
162
|
+
- **Clustering:** DBSCAN over cosine distance, `eps=0.35`, `min_samples=5`. Tune via `lensgrep recluster --eps 0.4 --min-samples 8`.
|
|
163
|
+
- **LoRA selection:** Greedy farthest-point sampling on embeddings — maximizes diversity. Quality scoring weights sharpness (Laplacian variance), face area, detection confidence.
|
|
164
|
+
|
|
165
|
+
## Why this exists
|
|
166
|
+
|
|
167
|
+
In 2026, the photo-management landscape has two extremes:
|
|
168
|
+
|
|
169
|
+
- **Bloated cloud apps** that ingest your library and "do AI on it": Google Photos, Apple Photos, Amazon Photos. Hand over your private life.
|
|
170
|
+
- **Self-hosted galleries with face features bolted on**: PhotoPrism, Immich, digiKam. Powerful but heavy — Docker, databases, web UIs, hours of setup, weeks of curating.
|
|
171
|
+
|
|
172
|
+
Neither helps if you just want to answer one question:
|
|
173
|
+
|
|
174
|
+
> "Where are all the photos of [person] in this big folder?"
|
|
175
|
+
|
|
176
|
+
…or:
|
|
177
|
+
|
|
178
|
+
> "Why does this folder weigh 80 GB? How many of these photos are actual duplicates?"
|
|
179
|
+
|
|
180
|
+
…or:
|
|
181
|
+
|
|
182
|
+
> "I'm training a Flux LoRA of my partner. Give me 50 diverse, high-quality photos."
|
|
183
|
+
|
|
184
|
+
That's the niche. **One job. Done well.**
|
|
185
|
+
|
|
186
|
+
## Comparison
|
|
187
|
+
|
|
188
|
+
| | lensgrep | PhotoPrism | Immich | digiKam | face_recognition |
|
|
189
|
+
|---|---|---|---|---|---|
|
|
190
|
+
| CLI-first | ✅ | ❌ | ❌ | ❌ | ✅ (library) |
|
|
191
|
+
| No Docker / DB setup | ✅ | ❌ | ❌ | ✅ | ✅ |
|
|
192
|
+
| Duplicate detection (exact + perceptual) | ✅ | partial | partial | partial | ❌ |
|
|
193
|
+
| LoRA dataset export | ✅ | ❌ | ❌ | ❌ | ❌ |
|
|
194
|
+
| FAISS vector search | ✅ | ❌ | ✅ | ❌ | ❌ |
|
|
195
|
+
| Modern face model (ArcFace) | ✅ | ✅ | ✅ | ❌ (older) | ❌ (dlib) |
|
|
196
|
+
| Local-only / no cloud | ✅ | ✅ | ✅ | ✅ | ✅ |
|
|
197
|
+
| Install in 10 seconds | ✅ | ❌ | ❌ | ❌ | ✅ |
|
|
198
|
+
|
|
199
|
+
## Privacy
|
|
200
|
+
|
|
201
|
+
Everything stays on your machine. No telemetry, no analytics, no network calls except the one-time model download from GitHub releases on first run. The SQLite database is plain text — you can `sqlite3 lensgrep.db ".dump"` and read it.
|
|
202
|
+
|
|
203
|
+
If you want to verify: `pip install lensgrep` ships ~600 lines of pure Python source. Audit it in an afternoon.
|
|
204
|
+
|
|
205
|
+
## Performance
|
|
206
|
+
|
|
207
|
+
Rough numbers on a 2020 laptop (Quadro RTX 5000 mobile, 16 GB VRAM):
|
|
208
|
+
|
|
209
|
+
| Library size | CPU scan | GPU scan (CUDA) |
|
|
210
|
+
|---|---|---|
|
|
211
|
+
| 1 000 photos | ~3 min | ~30 s |
|
|
212
|
+
| 10 000 photos | ~30 min | ~5 min |
|
|
213
|
+
| 100 000 photos | ~5 h | ~50 min |
|
|
214
|
+
|
|
215
|
+
Scan is the only slow part. After indexing, `find` / `export` / `lora` are sub-second on hundreds of thousands of faces.
|
|
216
|
+
|
|
217
|
+
CPU works fine if you're willing to leave it running overnight once. GPU setup needs CUDA Toolkit 12 + cuDNN 9 (`pip install "lensgrep[gpu]"` is necessary but not sufficient — see [docs/gpu.md](docs/gpu.md)).
|
|
218
|
+
|
|
219
|
+
## Roadmap
|
|
220
|
+
|
|
221
|
+
- [x] CLI MVP (scan, people, name, find, export, lora, dupes, stats)
|
|
222
|
+
- [ ] Per-person nearest-neighbor add: `lensgrep find-similar <photo.jpg>`
|
|
223
|
+
- [ ] EXIF date timeline filter: `lensgrep find Mom --year 2024`
|
|
224
|
+
- [ ] EXIF GPS map output
|
|
225
|
+
- [ ] HEIC support
|
|
226
|
+
- [ ] `lensgrep serve` — read-only web UI (FastAPI + minimal frontend) for browsing
|
|
227
|
+
- [ ] BLIP / Florence-2 auto-captions for richer LoRA training data
|
|
228
|
+
|
|
229
|
+
If the project gains traction, the web UI is next. Open an issue if you want to push it forward.
|
|
230
|
+
|
|
231
|
+
## Contributing
|
|
232
|
+
|
|
233
|
+
PRs welcome. Project conventions:
|
|
234
|
+
|
|
235
|
+
- Ruff for lint + format (`ruff check . && ruff format .`)
|
|
236
|
+
- Pytest for tests (`pytest`)
|
|
237
|
+
- Conventional commits (`feat:`, `fix:`, `docs:`, `refactor:`)
|
|
238
|
+
|
|
239
|
+
```bash
|
|
240
|
+
git clone https://github.com/Kris-O/lensgrep
|
|
241
|
+
cd lensgrep
|
|
242
|
+
pip install -e ".[dev]"
|
|
243
|
+
pytest
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
## License
|
|
247
|
+
|
|
248
|
+
MIT — see [LICENSE](LICENSE).
|
|
249
|
+
|
|
250
|
+
## Acknowledgments
|
|
251
|
+
|
|
252
|
+
Standing on the shoulders of:
|
|
253
|
+
- [InsightFace](https://github.com/deepinsight/insightface) — face detection + recognition
|
|
254
|
+
- [FAISS](https://github.com/facebookresearch/faiss) — vector search
|
|
255
|
+
- [Typer](https://github.com/fastapi/typer) + [Rich](https://github.com/Textualize/rich) — joy of CLI dev
|
|
256
|
+
|
|
257
|
+
Inspired by the question: *why isn't this just one command yet?*
|