imgad 0.1.0__tar.gz

imgad-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+

imgad-0.1.0/PKG-INFO

@@ -0,0 +1,434 @@
+Metadata-Version: 2.4
+Name: imgad
+Version: 0.1.0
+Summary: Image anomaly detection with foundation-model features (CLIP, SigLIP, DINOv2) + classical detectors, Anomalib models, and statistical benchmarking
+Author: Jónathan Heras
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/joheras/imgad
+Project-URL: Repository, https://github.com/joheras/imgad
+Project-URL: Issues, https://github.com/joheras/imgad/issues
+Keywords: anomaly-detection,one-class,computer-vision,foundation-models,clip,dinov2,anomalib,patchcore,mvtec,benchmark
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Image Recognition
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.23
+Requires-Dist: scipy>=1.9
+Requires-Dist: scikit-learn>=1.2
+Requires-Dist: torch>=2.0
+Requires-Dist: transformers>=4.40
+Requires-Dist: pillow>=9.0
+Requires-Dist: joblib>=1.2
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: tqdm>=4.60
+Provides-Extra: open-clip
+Requires-Dist: open_clip_torch>=2.20; extra == "open-clip"
+Provides-Extra: pyod
+Requires-Dist: pyod>=1.1; extra == "pyod"
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.5; extra == "pandas"
+Provides-Extra: anomalib
+Requires-Dist: anomalib>=2.0; extra == "anomalib"
+Provides-Extra: report
+Requires-Dist: matplotlib>=3.5; extra == "report"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: matplotlib>=3.5; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Provides-Extra: all
+Requires-Dist: open_clip_torch>=2.20; extra == "all"
+Requires-Dist: pyod>=1.1; extra == "all"
+Requires-Dist: pandas>=1.5; extra == "all"
+Requires-Dist: anomalib>=2.0; extra == "all"
+Requires-Dist: matplotlib>=3.5; extra == "all"
+Requires-Dist: pytest>=7; extra == "all"
+Dynamic: license-file
+
+# imgad — Image anomaly detection with foundation models, classical detectors and Anomalib
+
+[![CI](https://github.com/joheras/imgad/actions/workflows/ci.yml/badge.svg)](https://github.com/joheras/imgad/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/imgad.svg)](https://pypi.org/project/imgad/)
+[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/joheras/imgad/blob/main/LICENSE)
+
+A modular framework for image-level anomaly detection in the **one-class** setting (training uses
+normal images only):
+
+```
+images ──► FeatureExtractor ──► preprocessing ──► classical detector ──► anomaly score
+          (CLIP, SigLIP, DINOv2…   (l2, PCA,         (kNN, Mahalanobis,     (+ calibrated threshold)
+           any HF model)            standard…)        LOF, OCSVM, IF…)
+```
+
+It also wraps end-to-end [Anomalib](https://github.com/open-edge-platform/anomalib) models (PatchCore,
+PaDiM, EfficientAD…) with pixel-level anomaly maps, and includes a benchmarking module with repeated
+runs, confidence intervals, statistical tests and automatic markdown reports.
+
+## Installation
+
+```bash
+pip install imgad                    # core
+pip install "imgad[anomalib]"        # + Anomalib models (PatchCore, PaDiM, EfficientAD…)
+pip install "imgad[report]"          # + figures in benchmark reports (matplotlib)
+pip install "imgad[all]"             # + open_clip, pyod, pandas, anomalib, matplotlib, pytest
+```
+
+From source: `git clone https://github.com/joheras/imgad && pip install -e "imgad[dev]"`.
+Python ≥ 3.10.
+
+## Quick start
+
+```python
+from imgad import AnomalyDetectionPipeline, HFFeatureExtractor, build_detector, data
+from imgad.evaluation import evaluate_scores
+
+ds = data.mvtec_ad("/path/to/mvtec_anomaly_detection", "bottle")
+
+pipe = AnomalyDetectionPipeline(
+    extractor=HFFeatureExtractor("openai/clip-vit-base-patch16"),
+    detector=build_detector("knn", k=1),
+    preprocessing=["l2"],
+    cache_dir=".feature_cache",          # caches features per image
+)
+pipe.fit(ds.train_paths)                 # normal images only
+scores = pipe.score(ds.test_paths)       # higher = more anomalous
+preds  = pipe.predict(ds.test_paths)     # 0/1 with the calibrated threshold
+print(evaluate_scores(ds.test_labels, scores, preds))
+
+pipe.save("bottle.joblib")               # saves detector + config (not the model weights)
+pipe = AnomalyDetectionPipeline.load("bottle.joblib")
+```
+
+## Inference with a trained model
+
+The `.joblib` file stores the fitted detector, the preprocessing, the threshold, the extractor
+configuration and metadata (number of training images, date, threshold origin). Hugging Face weights
+are not stored: they are reloaded from the Hub (or the local HF cache) on `load`.
+
+```python
+from imgad import AnomalyPredictor
+
+predictor = AnomalyPredictor.load("bottle.joblib")   # device auto (cuda/mps/cpu); device="cpu" to force it
+r = predictor.predict_one("new.png")                 # path, PIL.Image or numpy array
+r.is_anomaly, r.score, r.normal_percentile
+
+results = predictor.predict("folder/")               # folder, list of paths/images, or a mix
+predictor.save_results(results, "preds.csv")         # .csv or .json
+for r in predictor.predict_iter("huge_folder/", chunk_size=256): ...   # constant memory
+```
+
+Each `Prediction` contains:
+
+| Field | Meaning |
+|---|---|
+| `score` | raw score (detector-dependent scale; higher = more anomalous) |
+| `is_anomaly` | `score > threshold` |
+| `threshold` | threshold used |
+| `normal_percentile` | fraction of normal training images with a lower score (0–1). Comparable across detectors: ~0.5 = typical normal image, 1.0 = rarer than any normal image seen |
+
+**Tune the threshold** on a labelled validation set (no retraining):
+
+```python
+pipe = AnomalyDetectionPipeline.load("bottle.joblib")
+pipe.calibrate_threshold(val_paths, val_labels, method="f1")   # f1 | youden | fpr | tpr
+pipe.save("bottle_cal.joblib")
+# Normal images only: method="fpr", target_fpr=0.01  -> ≤1% false alarms
+```
+
+Or on the fly: `AnomalyPredictor.load("bottle.joblib", threshold=0.02)`.
+
+Shortcuts: the extractor can be an HF id (`"facebook/dinov2-base"`) and the detector a name
+(`"mahalanobis"`) or a dict (`{"name": "knn", "k": 5}`).
+
+## Feature extractors
+
+`HFFeatureExtractor(model_name, pooling="auto", layer=None, normalize=True, ...)` works with any
+vision model on the Hub that loads with `AutoModel`:
+
+| Family | Examples | Default pooling |
+|---|---|---|
+| CLIP-style dual encoders | `openai/clip-*`, `google/siglip-*`, `google/siglip2-*`, `facebook/metaclip-*`, `kakaobrain/align-base`, `laion/CLIP-*` (HF format) | `projection` → `get_image_features` |
+| Vision backbones | `facebook/dinov2-*`, `google/vit-*`, `microsoft/beit-*`, `facebook/convnext-*`, `microsoft/resnet-*` | `pooler` |
+
+`pooling` options: `auto`, `projection`, `pooler`, `cls`, `mean` (mean of patch tokens), `cls+mean`.
+`layer` selects intermediate layers (`layer=-4`, or `layer=[6, 9, 12]` to concatenate several), which
+often capture local defects better than the more semantic last layer.
+
+For **open_clip** checkpoints (`hf-hub:laion/...`, EVA-CLIP, DataComp…):
+`OpenCLIPFeatureExtractor("hf-hub:laion/CLIP-ViT-B-32-laion2B-s34B-b79K")`.
+
+## Detectors
+
+All detectors share the same API (`fit`, `score`, `predict`) and the convention **higher score = more
+anomalous**.
+
+| Name | Method | Notes |
+|---|---|---|
+| `knn` | Distance to the k nearest normal samples | Very strong baseline with CLIP/DINO; `metric="cosine"` |
+| `mahalanobis` / `gaussian` | Mahalanobis distance (Ledoit-Wolf, OAS, MCD) | Very good with foundation-model features |
+| `centroid` | Distance to the mean embedding | Minimal cost |
+| `kmeans` | Distance to the nearest centroid | Multimodal normality |
+| `lof` | Local Outlier Factor (novelty mode) | |
+| `gmm` | Negative log-likelihood under a Gaussian mixture | |
+| `kde` | Negative KDE log-density | Better after `pca:<n>` to few dimensions |
+| `ocsvm` | One-Class SVM | |
+| `iforest` | Isolation Forest | |
+| `pca` | PCA reconstruction error | |
+| `ensemble` | Mean of normalised detectors (z-score or rank) | |
+| `sklearn` | Wraps any sklearn estimator | `SklearnDetector(estimator=EllipticEnvelope())` |
+| `pyod` | Any PyOD model | `{"name": "pyod", "model": "ECOD"}` |
+
+**Threshold**: after `fit`, `threshold_` is set to the `threshold_quantile` (0.95 by default) of
+*out-of-sample* scores of the normal data: leave-one-out for kNN/LOF and 5-fold cross-validation for
+the rest. This avoids the optimistic threshold of detectors that overfit the training set (KDE, OCSVM,
+PCA…). It can be disabled with `det.calibration = "train"`.
+
+## Anomalib models (with anomaly maps)
+
+`AnomalibPipeline` wraps any **one-class** [Anomalib](https://github.com/open-edge-platform/anomalib)
+model (`patchcore`, `padim`, `efficient_ad`, `fastflow`, `stfpm`, `reverse_distillation`, `cflow`,
+`draem`…; `imgad list` shows them) with the same API as the feature pipelines, and adds **pixel-level
+anomaly maps**:
+
+```python
+from imgad import AnomalibPipeline, evaluate_maps
+
+pc = AnomalibPipeline("patchcore", backbone="wide_resnet50_2", layers=["layer2", "layer3"], image_size=256)
+pc.fit(ds.train_paths)                              # normal images only
+scores, maps = pc.score_and_maps(ds.test_paths)     # maps: list of arrays (model resolution)
+evaluate_maps(ds.test_masks, maps)                  # pixel_auroc, pixel_aupr, pixel_f1_max, aupro
+pc.predict_masks(maps)                              # binary masks (map > pixel_threshold_)
+pc.save("bottle_pc.joblib")                         # loaded by AnomalyPredictor.load / imgad predict as usual
+```
+
+- Anomalib provides the models and the training loop; its post-processing (normalisation and threshold
+  learnt on a labelled validation set) is disabled. Scores and maps are the **raw** model outputs
+  (higher = more anomalous).
+- **Threshold**: as for the detectors, the `threshold_quantile` (0.95) of the scores of *unseen* normal
+  images: `calibration_split=0.2` holds out that 20% of the training images (not used for training).
+  `calibration_split=0` uses training scores (optimistic, especially for PatchCore). The pixel threshold
+  (`pixel_threshold_`) is the same quantile of the maximum of each held-out normal map; with validation
+  masks it can be tuned with `calibrate_pixel_threshold(images, masks)` (pixel F1).
+  `calibrate_threshold` works as in the other pipelines.
+- Images are resized to `image_size` **without cropping**, so the maps cover the whole image.
+  `score_and_maps(..., size="original")` returns them at each image's own size.
+- The `.joblib` stores the configuration **and the weights** (e.g. the PatchCore memory bank): nothing
+  needs to be downloaded to load it.
+- Model kwargs are those of the Anomalib constructor (`backbone`, `layers`, `coreset_sampling_ratio`,
+  `pre_trained`…). For models trained over several epochs: `max_epochs` or
+  `trainer_kwargs={"max_steps": ...}`. EfficientAD trains with batch size 1 and downloads imagenette.
+- One-class models only; zero/few-shot models (WinCLIP, AnomalyDINO…) are not supported.
+
+Inference with maps:
+
+```python
+predictor = AnomalyPredictor.load("bottle_pc.joblib")
+res = predictor.predict("folder/", return_maps=True)   # r.anomaly_map (original size), r.anomalous_area
+predictor.save_maps(res, "maps/")                      # <folder>_<name>_heatmap.png + _mask.png
+predictor.save_results(res, "preds.csv")               # includes the heatmap and mask paths
+```
+
+Heatmaps use a fixed scale per model (blue = typical normal, middle of the scale = pixel threshold,
+red = far above it), so normal images look blue instead of being rescaled.
+
+## Preprocessing
+
+List of steps applied before the detector (fitted on the normal data):
+`"l2"`, `"standard"`, `"center"`, `"pca:0.95"` (explained variance) or `"pca:64"` (components),
+`"pca_whiten:<n>"`, or any sklearn transformer.
+
+## Data
+
+```python
+data.mvtec_ad(root, "screw")                     # MVTec AD / MVTec layout (converted VisA, etc.)
+data.from_folders(train_normal, test_normal, [test_anom_1, test_anom_2])
+data.one_vs_rest(root, normal_class="cat")       # semantic AD from a classification dataset
+```
+
+Each returns an `AnomalyDataset` with `train_paths`, `test_paths`, `test_labels`, `test_types` and, for
+MVTec, `test_masks` (path of the ground-truth mask in `ground_truth/<type>/<name>_mask.png`, or `None`
+for normal images).
+
+## Evaluation and benchmarks
+
+`evaluate_scores` returns AUROC, AUPR, F1-max and FPR@95%TPR (and F1/TPR/FPR at the threshold if
+predictions are given). `per_type_auroc` breaks the AUROC down by defect type.
+
+`evaluate_maps(masks, maps)` gives the localisation metrics: pixel-AUROC, pixel-AUPR, pixel-F1-max and
+AUPRO (FPR ≤ 0.3, the standard MVTec AD metric). Masks are resized to each map's resolution.
+
+`benchmark(extractors, detectors, datasets, preprocessings, anomalib_models=[...])` evaluates the full
+grid, extracting features once per (extractor, dataset), plus each Anomalib model per dataset (with
+pixel metrics when masks are available). In YAML, use the `anomalib:` key (see
+`configs/mvtec_benchmark.yaml`).
+
+## Benchmark reports
+
+`imgad benchmark --report DIR` writes a markdown report with figures:
+
+```bash
+imgad benchmark --config configs/mvtec_benchmark.yaml --report report/ --seeds 0 1 2 --bootstrap 1000
+```
+
+```
+report/
+  report.md       # the report
+  figures/*.png   # every figure referenced in the report
+  results.csv     # one row per dataset × configuration × seed
+  config.yaml     # configuration used
+```
+
+The report contains:
+
+| Section | Content |
+|---|---|
+| Summary | best configuration, runner-up and significance, best extractors/detectors, hardest dataset |
+| Data analysis | images per split, defect types, resolution, masks; example images with ground-truth masks; warnings (small or imbalanced test sets, few training images) |
+| Methods | extractors (with dimension), preprocessing, detectors and Anomalib models with a short description; threshold protocol, metrics, repetitions and statistical tests |
+| Results | ranking table and chart, extractor × detector heatmap, per-dataset results, ROC curves, score histograms with the threshold, per-defect-type AUROC, operating point, pixel-level metrics and example anomaly maps, spread across seeds, Friedman/Nemenyi/Wilcoxon (or DeLong) with a critical-difference diagram, computational cost |
+| Discussion | extractor vs. detector effect, consistency across datasets, significance, hard defect types, Anomalib vs. global features, threshold calibration, localisation |
+| Limitations and reproducibility | caveats derived from the setup, exact command, configuration and environment versions |
+
+Figures need matplotlib (`pip install "imgad[report]"`); without it the report contains tables only.
+The summary and discussion are generated from the numbers with simple rules: treat them as a first
+draft to check and complete. To rebuild a report without rerunning the benchmark:
+
+```bash
+imgad benchmark --config cfg.yaml --scores-out scores.joblib
+imgad report --scores scores.joblib --config cfg.yaml --out report/ --title "My benchmark"
+```
+
+From Python: `imgad.report.generate_report(rows, datasets, "report/", config=cfg)` with
+`rows = benchmark(..., keep_scores=True)`.
+
+## Statistical analysis: repetitions, confidence intervals and tests
+
+**Repetitions.** `benchmark(..., seeds=[0, 1, 2, 3, 4], train_subsample=0.8)` runs every configuration
+once per seed (column `seed`). The seed sets the detectors' `random_state` (unless the spec fixes it),
+the calibration folds, the Anomalib seed and the **training subset** (`train_subsample`: a fraction if
+float, a number of images if int). Without subsampling, deterministic detectors (kNN, Mahalanobis…)
+give the same result for every seed. Features are extracted once and reused across seeds.
+
+**Confidence intervals.** There are two sources of uncertainty, measured separately:
+
+| Source | How | Where |
+|---|---|---|
+| Test sample | stratified bootstrap of the test images (`bootstrap=1000`) | `auroc_ci_low/high`, `aupr_ci_low/high` in each row |
+| Training | repetitions with seeds + `train_subsample` | `stats.summarize_seeds(rows)`: mean, standard deviation and t-based CI of the mean |
+
+**Statistical tests** (`imgad.stats`):
+
+| Question | Test | Function |
+|---|---|---|
+| Does A have a higher AUROC than B **on this dataset**? | DeLong (correlated AUROCs, same images) | `delong_test(y, sa, sb)`, `delong_vs_baseline(rows, base)` |
+| Same for AUPR, F1-max or another metric | paired bootstrap | `paired_bootstrap_test(y, sa, sb, metric="aupr")` |
+| Are there differences between configurations **across datasets**? | Friedman (+ Iman-Davenport correction) | `compare_configs(rows)` |
+| Which pairs differ? | Nemenyi (critical difference of ranks) and Wilcoxon signed-rank + Holm | `compare_configs(rows)["pairs"]`, `plot_cd_diagram` |
+
+To compare across datasets, seeds are first averaged within each dataset, and each dataset (e.g. each
+MVTec category) is a block, following Demšar (2006). Wilcoxon-Holm usually has more power with few
+configurations, while Nemenyi is the classical post-hoc test after Friedman and the one drawn in the CD
+diagram. DeLong needs per-image scores (`keep_scores=True`).
+
+```python
+from imgad import benchmark, stats
+
+rows = benchmark(extractors, detectors, datasets, seeds=range(5), train_subsample=0.8,
+                 bootstrap=1000, keep_scores=True, anomalib_models=["patchcore"])
+stats.summarize_seeds(rows)                          # mean ± std per (dataset, config)
+res = stats.compare_configs(rows, metric="auroc")    # also aupr, pixel_auroc, aupro, fpr@95tpr…
+print(stats.report_markdown(res))                    # mean ranks, Friedman, CD, pairs with Holm
+stats.plot_cd_diagram(res, "cd.png")
+stats.delong_vs_baseline(rows, baseline=res["configs"][0])
+```
+
+From the CLI:
+
+```bash
+imgad benchmark --config configs/mvtec_benchmark.yaml --seeds 0 1 2 3 4 --train-subsample 0.8 \
+                --bootstrap 1000 --out results.csv --scores-out scores.joblib --stats
+imgad stats --results results.csv --metric auroc --cd-plot cd.png --out stats.md
+imgad stats --scores scores.joblib --baseline "openai/clip-vit-base-patch16 | l2 | knn(k=1)"   # + DeLong
+```
+
+## CLI
+
+```bash
+imgad list
+imgad fit --model openai/clip-vit-base-patch16 --detector knn -p k=1 --prep l2 \
+          --train data/bottle/train/good --out bottle.joblib
+
+# inference: images and/or folders; output to stdout, .csv or .json
+imgad predict --model bottle.joblib new.png data/bottle/test --out preds.csv
+imgad predict --model bottle.joblib folder/ --only-anomalies --threshold 0.02 --device cpu
+
+# re-tune the threshold on labelled validation data (f1 | youden | fpr | tpr)
+imgad calibrate --model bottle.joblib --normal val/good --anomalous val/defects --method f1 --out bottle_cal.joblib
+imgad info --model bottle_cal.joblib         # configuration, threshold and metadata
+
+imgad benchmark --config configs/mvtec_benchmark.yaml --out results.csv --report report/
+
+# Anomalib models: -p sets the model kwargs; predict saves heatmaps and masks
+imgad fit --anomalib patchcore -p backbone=wide_resnet50_2 "layers=[layer2,layer3]" \
+          --image-size 256 --train data/bottle/train/good --out bottle_pc.joblib
+imgad predict --model bottle_pc.joblib data/bottle/test --maps-dir maps/ --out preds.csv
+```
+
+## Extending
+
+Registering your own detector or extractor makes it available by name in Python, YAML and the CLI (see
+`examples/extending.py`):
+
+```python
+from imgad import BaseDetector, register_detector
+
+@register_detector("my_detector")
+class MyDetector(BaseDetector):
+    def _fit(self, X): ...
+    def _score(self, X): ...   # higher = more anomalous
+```
+
+## Example results
+
+[`benchmarks/mvtec_4categories/`](https://github.com/joheras/imgad/tree/main/benchmarks/mvtec_4categories) contains a benchmark of 5 extractors
+× 11 detectors on 4 MVTec AD categories (report, results CSV and configuration).
+
+## Project layout
+
+```
+src/imgad/
+  features/      base.py (interface, registry, cache) · huggingface.py · open_clip.py
+  detectors/     base.py (interface, registry, calibration) · classical.py
+  anomalib_backend.py (AnomalibPipeline) · maps.py (masks, heatmaps)
+  preprocessing.py · pipeline.py · inference.py · data.py · evaluation.py · stats.py · report.py · cli.py
+configs/         example benchmark YAML files
+examples/        quickstart.py · inference.py · compare_models.py · extending.py · anomalib_models.py
+benchmarks/      example benchmark results
+tests/           pytest (offline: a tiny CLIP generated locally; untrained ResNet-18 for Anomalib)
+```
+
+## Possible next steps
+
+- **Zero-/few-shot** detection with text prompts (WinCLIP, AnomalyCLIP) using the text encoder.
+- Native PatchCore on patch features of foundation models (CLIP/DINOv2), to compare with Anomalib's (CNN).
+- Test-time augmentation (TTA) and multi-scale aggregation.
+- Few-shot sweep (`train_subsample` = 1, 2, 4, 8… images) reusing the seeds and the tests in `imgad.stats`.
+
+## License
+
+Distributed under the [Apache 2.0](https://github.com/joheras/imgad/blob/main/LICENSE) license.
+
+## Acknowledgements
+
+This work is partially supported by Agencia de Desarrollo Económico de La Rioja [ADER 2022-I-IDI-00015].