iaa-od 4.1.1__tar.gz

iaa_od-4.1.1/.gitignore

@@ -0,0 +1,48 @@
+# Optimized files
+__pycache__/
+
+# macOS File system
+.DS_Store
+
+# Image folder
+img/
+
+# Python virtual environment
+venv/
+
+# Jupyter Notebooks
+*.ipynb
+
+# Jupyter Notebook checkpoints
+.ipynb_checkpoints/
+
+# LSP files
+.virtual_documents/
+
+# Jetbrains IDEs
+.idea/
+
+# Package install files
+*.egg-info/
+
+# Old Ground Truth files
+data/gts/old/
+data/gts/old/**
+
+# Test scripts
+test*.py
+save_imgs.py
+
+# Output folder
+out/
+out/**
+
+# Distribution files
+dist/**
+
+# Other
+img_nums.txt
+
+# Sphinx build files
+docs/build/
+docs/source/generated/

iaa_od-4.1.1/.gitlab-ci.yml

@@ -0,0 +1,33 @@
+stages:
+  - docs
+  - publish
+
+pages:
+  stage: docs
+  image: python:3.10-slim
+  before_script:
+    - pip install --quiet -r docs/requirements.txt
+    - pip install --quiet -e .
+  script:
+    - sphinx-build -b html docs/source public
+  artifacts:
+    paths:
+      - public
+    expire_in: 1 week
+  rules:
+    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
+      changes:
+        - "docs/**/*"
+        - "src/iaa_od/**/*"
+        - "pyproject.toml"
+
+  publish-pypi:
+    stage: publish
+    image: python:3.10-slim
+    rules:
+      - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+/
+    before_script:
+      - pip install --upgrade build twine
+    script:
+      - python -m build
+      - twine upload dist/* -u __token__ -p "$PYPI_TOKEN"

iaa_od-4.1.1/CONTRIBUTING.md

@@ -0,0 +1,38 @@
+# Contributing to iaa-od
+
+Thank you for your interest in `iaa-od`. Contributions are welcome, whether they come from researchers applying the package, developers improving its implementation, or anyone who finds a bug.
+
+## Reporting a bug
+
+Please open a [GitLab issue](../../issues) and include:
+
+- your Python version and `iaa-od` version (`pip show iaa-od`)
+- a **minimal reproducible example** — the smallest snippet of code that triggers the problem
+- the full traceback, if applicable
+
+Before opening a new issue, check whether the problem has already been reported.
+
+## Suggesting a feature or change
+
+Open an issue describing what you have in mind before writing any code. This is especially important for changes that touch the core metrics or the disagreement decomposition — since this package accompanies a published methodology, modifications to the core algorithms need to be discussed first to ensure consistency with the paper.
+
+For smaller improvements (documentation, performance, new utilities that do not alter existing behaviour), a direct merge request is fine.
+
+## Submitting a merge request
+
+1. Fork the repository and create a branch from `master` with a descriptive name (e.g. `fix/clustering-edge-case` or `feat/new-visualisation`).
+2. Make your changes. If you are fixing a bug, a short comment explaining the root cause is appreciated.
+3. Ensure the package still builds cleanly (`python -m build`) and imports correctly.
+4. Open a merge request against `master` with a clear description of what was changed and why.
+
+## Scope
+
+`iaa-od` implements the metrics described in the accompanying paper. Contributions that extend the package — new metrics, new dataset formats, additional error types — are welcome, but proposals that would change the definition or behaviour of existing metrics should be raised as an issue first. This keeps the package consistent with the published work and avoids silent incompatibilities between versions.
+
+## Code style
+
+The codebase uses standard Python conventions. There is no enforced formatter at this time; just try to match the style of the surrounding code.
+
+## Questions
+
+If you are unsure whether something is a bug or a feature, or want to discuss the methodology, feel free to open an issue with the `question` label.

iaa_od-4.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Mario Merlo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

iaa_od-4.1.1/PKG-INFO

@@ -0,0 +1,115 @@
+Metadata-Version: 2.4
+Name: iaa-od
+Version: 4.1.1
+Summary: A Python package for analysing object detection datasets through Inter-Annotatator Agreement (IAA) metrics.
+Project-URL: Homepage, https://gitlab.com/MrVideo/iaa.py
+Author-email: Mario Merlo <mario.merlo@polimi.it>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: annotation quality,dataset analysis,inter-annotator agreement,object detection
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: matplotlib>=3.10.6
+Requires-Dist: mplcursors>=0.7
+Requires-Dist: numpy>=2.2.6
+Requires-Dist: scikit-learn>=1.7.2
+Requires-Dist: scipy>=1.15.3
+Provides-Extra: dev
+Requires-Dist: basedpyright>=1.32.1; extra == 'dev'
+Requires-Dist: jupyter-lsp>=2.3.0; extra == 'dev'
+Requires-Dist: jupyterlab-lsp>=5.2.0; extra == 'dev'
+Requires-Dist: jupyterlab>=4.4.10; extra == 'dev'
+Requires-Dist: jupytext>=1.18.1; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# `iaa-od`: a comprehensive IAA analysis suite
+
+This package provides a comprehensive Inter-Annotator Agreement analysis suite for Object Detection tasks.
+
+## Installation
+
+You can install the `iaa-od` package by cloning this repository and then running the following command on its top level:
+
+```
+pip install .
+```
+
+### Development installation
+
+If you want to make modifications to the package, you may install it like so:
+
+```
+pip install -e .
+```
+
+If you wish to do some testing through Jupyter Notebooks, install the optional dependencies as well:
+
+```
+pip install -e ".[dev]"
+```
+
+## Computed metrics
+
+Below is a list of currently implemented metrics.
+
+### Krippendorff's $\alpha$
+
+Krippendorff's $\alpha$ coefficient is defined as:
+
+$$\alpha = 1 - \frac{D_o}{D_e}$$
+
+and is computed in our package according to [this paper by Klaus Krippendorff](https://repository.upenn.edu/handle/20.500.14332/2089).
+
+The *units* in each image are found through agglomerative clustering.
+
+### Rater vitality
+
+[Rater vitality](10.1109/SocialSens.2018.00018) measures the contribution of each annotator to the global agreement value. It is computed as:
+
+$$\nu_a = \alpha - \alpha_a$$
+
+where $\alpha_a$ is the value of Krippendorff's $\alpha$ computed excluding annotator $a$ from the dataset.
+
+### Scale complexity
+
+[Scale complexity](10.1109/SocialSens.2018.00018) measures the ambiguity found across categories of the same dataset by grouping them into user-defined macro-categories. It is computed as:
+
+$$\text{SC} = \alpha_M - \alpha$$
+
+where $\alpha_M$ is the value of Krippendorff's $\alpha$ computed on the macro-categories. If $\text{SC} > 0$, categories grouped in the same macro-categories are ambiguous.
+
+### L-Score (Localisation Score)
+
+L-Score measures the localisation agreement reached by the annotators by only considering the locations of their bounding boxes. It is computed as:
+
+$$L_{\text{score}} = \frac{|A|}{2|A|-1} \left[ \sum_{i=2}^{|A|} \left(\frac{i-1}{|A|}\, p_i\right) - p_1 + 1\right]$$
+
+L-Score $\in [0,1]$, where 1 indicates perfect localisation agreement.
+
+### C-Score (Categorisation Score)
+
+C-Score measures the categorisation agreement reached by annotators by analysing intersecting bounding boxes and comparing their labels.
+
+Computation is not reported here for brevity.
+
+C-Score $\in [0,1]$, where 1 indicates perfect categorisation agreement.
+
+### Count and size granularity
+
+Count and size granularity measure the behaviour of annotators when annotating large groups of object:
+
+- an annotator who groups many instances into a single bounding box annotates "by group";
+- an annotator who creates several bounding boxes for each instance annotates "by instance".
+
+Computation of these metrics is not reported here for brevity.
+
+Count and size granularities both rely on a benchmark annotator, found during computation, to whom all other annotators are compared to.
+
+## Working example
+
+`main.py` provides a working example of the package by performing all available analyses on a dataset with three annotators. More examples are also provided in the `experiments` directory: specifically, `experiments/images.py` shows how to utilise the `visualisation` module.
+
+## Jupyter notebooks
+
+The `notebooks` directory contains a number of Jupyter notebooks to play around with data.

iaa_od-4.1.1/README.md

@@ -0,0 +1,90 @@
+# `iaa-od`: a comprehensive IAA analysis suite
+
+This package provides a comprehensive Inter-Annotator Agreement analysis suite for Object Detection tasks.
+
+## Installation
+
+You can install the `iaa-od` package by cloning this repository and then running the following command on its top level:
+
+```
+pip install .
+```
+
+### Development installation
+
+If you want to make modifications to the package, you may install it like so:
+
+```
+pip install -e .
+```
+
+If you wish to do some testing through Jupyter Notebooks, install the optional dependencies as well:
+
+```
+pip install -e ".[dev]"
+```
+
+## Computed metrics
+
+Below is a list of currently implemented metrics.
+
+### Krippendorff's $\alpha$
+
+Krippendorff's $\alpha$ coefficient is defined as:
+
+$$\alpha = 1 - \frac{D_o}{D_e}$$
+
+and is computed in our package according to [this paper by Klaus Krippendorff](https://repository.upenn.edu/handle/20.500.14332/2089).
+
+The *units* in each image are found through agglomerative clustering.
+
+### Rater vitality
+
+[Rater vitality](10.1109/SocialSens.2018.00018) measures the contribution of each annotator to the global agreement value. It is computed as:
+
+$$\nu_a = \alpha - \alpha_a$$
+
+where $\alpha_a$ is the value of Krippendorff's $\alpha$ computed excluding annotator $a$ from the dataset.
+
+### Scale complexity
+
+[Scale complexity](10.1109/SocialSens.2018.00018) measures the ambiguity found across categories of the same dataset by grouping them into user-defined macro-categories. It is computed as:
+
+$$\text{SC} = \alpha_M - \alpha$$
+
+where $\alpha_M$ is the value of Krippendorff's $\alpha$ computed on the macro-categories. If $\text{SC} > 0$, categories grouped in the same macro-categories are ambiguous.
+
+### L-Score (Localisation Score)
+
+L-Score measures the localisation agreement reached by the annotators by only considering the locations of their bounding boxes. It is computed as:
+
+$$L_{\text{score}} = \frac{|A|}{2|A|-1} \left[ \sum_{i=2}^{|A|} \left(\frac{i-1}{|A|}\, p_i\right) - p_1 + 1\right]$$
+
+L-Score $\in [0,1]$, where 1 indicates perfect localisation agreement.
+
+### C-Score (Categorisation Score)
+
+C-Score measures the categorisation agreement reached by annotators by analysing intersecting bounding boxes and comparing their labels.
+
+Computation is not reported here for brevity.
+
+C-Score $\in [0,1]$, where 1 indicates perfect categorisation agreement.
+
+### Count and size granularity
+
+Count and size granularity measure the behaviour of annotators when annotating large groups of object:
+
+- an annotator who groups many instances into a single bounding box annotates "by group";
+- an annotator who creates several bounding boxes for each instance annotates "by instance".
+
+Computation of these metrics is not reported here for brevity.
+
+Count and size granularities both rely on a benchmark annotator, found during computation, to whom all other annotators are compared to.
+
+## Working example
+
+`main.py` provides a working example of the package by performing all available analyses on a dataset with three annotators. More examples are also provided in the `experiments` directory: specifically, `experiments/images.py` shows how to utilise the `visualisation` module.
+
+## Jupyter notebooks
+
+The `notebooks` directory contains a number of Jupyter notebooks to play around with data.