granuscore 1.0.0__tar.gz

granuscore-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Granuscore Authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

granuscore-1.0.0/MANIFEST.in

@@ -0,0 +1,5 @@
+prune assets
+prune evaluation
+prune training_scripts
+prune notebooks
+prune data

granuscore-1.0.0/PKG-INFO

@@ -0,0 +1,242 @@
+Metadata-Version: 2.4
+Name: granuscore
+Version: 1.0.0
+Summary: Granularity scoring for natural language
+Author-email: Lukas Ellinger <lukas.ellinger@tum.de>
+License-Expression: MIT
+Project-URL: Repository, https://github.com/lukasellinger/granuscore
+Project-URL: Issues, https://github.com/lukasellinger/granuscore/issues
+Keywords: granularity,natural language processing,nlp,text analysis,semantic abstraction,specificity
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: <3.13,>=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: faiss-cpu>=1.13.2
+Requires-Dist: hierarchy_transformers>=0.1.1
+Requires-Dist: lightgbm>=4.6.0
+Requires-Dist: spacy>=3.8.11
+Requires-Dist: torch>=2.9.1
+Requires-Dist: numpy>=1.26.4
+Requires-Dist: platformdirs>=3.10.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: tqdm>=4.66.0
+Requires-Dist: openai>=2.21.0
+Provides-Extra: dev
+Requires-Dist: optuna>=4.6.0; extra == "dev"
+Requires-Dist: pygam>=0.12.0; extra == "dev"
+Requires-Dist: python-dotenv>=1.2.1; extra == "dev"
+Requires-Dist: ollama>=0.6.1; extra == "dev"
+Requires-Dist: google-genai>=1.66.0; extra == "dev"
+Requires-Dist: wget>=3.2; extra == "dev"
+Dynamic: license-file
+
+[![PyPI version](https://img.shields.io/pypi/v/granuscore.svg)](https://pypi.org/project/granuscore/)
+[![GitHub](https://img.shields.io/badge/GitHub-Repository-black)](https://github.com/lukasellinger/granuscore)
+# Granuscore
+
+**Granuscore** is a Python library for measuring the *semantic granularity* of natural language text.
+
+It provides an end-to-end pipeline that:
+
+1. splits text into referential units,
+2. assigns continuous granularity scores to each unit,
+3. aggregates these scores into document-level estimates.
+
+Granuscore is designed for analyzing how *fine-grained* or *coarse-grained* textual expressions are in applications such as question answering, educational dialogue, summarization, and scientific writing.
+
+---
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip install granuscore
+```
+
+Or install the latest development version locally:
+
+```bash
+git clone https://github.com/lukasellinger/granuscore.git
+cd granuscore
+pip install -e .
+```
+
+Optional development dependencies:
+
+```bash
+pip install -e ".[dev]"
+```
+
+---
+
+## Quick Start
+
+```python
+from granuscore import GranuScore
+
+scorer = GranuScore()
+
+text = """
+Tony Hawk was born in San Diego.
+"""
+
+score = scorer(text)
+
+print(score)
+```
+
+By default, Granuscore returns percentile scores, where higher values correspond to coarser-grained expressions.
+
+---
+
+## Default Configuration
+
+The default configuration reproduces the setup used in the paper.
+
+```python
+scorer = GranuScore()
+```
+
+Equivalent to:
+
+```python
+scorer = GranuScore(
+    predictor_type="hit",
+)
+```
+
+Default settings:
+
+- `predictor_type="hit"`
+- `model_name="Hierarchy-Transformers/HiT-MiniLM-L12-WordNetNoun"`
+- `search_method="random_anchors"`
+- `random_anchors_k=999`
+
+Required artifacts such as:
+- FAISS indices,
+- anchor vectors,
+- LightGBM models,
+- and reference percentile distributions
+
+are automatically downloaded and cached on first use.
+
+---
+
+## Important Compatibility Note
+
+The default configuration works out of the box and is the recommended setup.
+
+If you customize components such as:
+- the embedding model,
+- search method,
+- FAISS index,
+- anchor vectors,
+- or LightGBM model,
+
+you must ensure that all resources are compatible with each other.
+
+For example, a LightGBM model trained using:
+
+```python
+search_method="random_anchors"
+```
+
+should not be combined with:
+
+```python
+search_method="nearest_neighbor"
+```
+
+Similarly, FAISS indices, anchor vectors, percentile reference distributions, and LightGBM models must originate from the same embedding space and training configuration.
+
+Compatibility between custom resources is not validated automatically.
+
+---
+
+## Notebook Tutorial
+
+An interactive introduction is available in:
+
+```text
+notebooks/getting_started.ipynb
+```
+
+---
+
+## Repository Structure
+
+```text
+granuscore/
+├── src/
+│   └── granuscore/
+│       ├── pipeline.py
+│       ├── granularity_predictor.py
+│       ├── claim_splitter.py
+│       ├── bucket_output.py
+│       ├── cache.py
+│       └── artifacts.py
+├── notebooks/
+│   └── getting_started.ipynb
+├── training_scripts/
+├── evaluation/
+├── assets/
+├── data/ (needs to be externally downloaded)
+├── pyproject.toml
+├── LICENSE
+└── README.md
+```
+
+---
+
+## Reproducing Paper Experiments
+
+The datasets and precomputed resources required to reproduce the experiments from the paper are available here:
+
+https://drive.google.com/drive/folders/1mJdUENOxHEiuYn-_f1KRQ1PZggXJDnb4?usp=sharing
+
+Download the archive and extract it into the repository root:
+
+```bash
+unzip data.zip
+```
+
+This will create the expected directory structure used by the training and evaluation scripts.
+
+---
+
+## Training Pipeline
+
+Training uses precomputed `.pkl` feature files.
+
+1. Generate precomputed datasets:
+
+```text
+training_scripts/build_precalc_data/
+```
+
+2. Train LightGBM models:
+
+```bash
+python training_scripts/train_lgb_models.py
+```
+
+---
+
+## Citation
+Updated Citation information will be added after publication.
+
+```bibtex
+@misc{ellinger2026granuscore,
+  title={Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering},
+  author={Ellinger, Lukas and Fichtl, Alexander M. and Anschütz, Miriam and Groh, Georg},
+  year={2026}
+}
+```

granuscore-1.0.0/README.md

@@ -0,0 +1,203 @@
+[![PyPI version](https://img.shields.io/pypi/v/granuscore.svg)](https://pypi.org/project/granuscore/)
+[![GitHub](https://img.shields.io/badge/GitHub-Repository-black)](https://github.com/lukasellinger/granuscore)
+# Granuscore
+
+**Granuscore** is a Python library for measuring the *semantic granularity* of natural language text.
+
+It provides an end-to-end pipeline that:
+
+1. splits text into referential units,
+2. assigns continuous granularity scores to each unit,
+3. aggregates these scores into document-level estimates.
+
+Granuscore is designed for analyzing how *fine-grained* or *coarse-grained* textual expressions are in applications such as question answering, educational dialogue, summarization, and scientific writing.
+
+---
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip install granuscore
+```
+
+Or install the latest development version locally:
+
+```bash
+git clone https://github.com/lukasellinger/granuscore.git
+cd granuscore
+pip install -e .
+```
+
+Optional development dependencies:
+
+```bash
+pip install -e ".[dev]"
+```
+
+---
+
+## Quick Start
+
+```python
+from granuscore import GranuScore
+
+scorer = GranuScore()
+
+text = """
+Tony Hawk was born in San Diego.
+"""
+
+score = scorer(text)
+
+print(score)
+```
+
+By default, Granuscore returns percentile scores, where higher values correspond to coarser-grained expressions.
+
+---
+
+## Default Configuration
+
+The default configuration reproduces the setup used in the paper.
+
+```python
+scorer = GranuScore()
+```
+
+Equivalent to:
+
+```python
+scorer = GranuScore(
+    predictor_type="hit",
+)
+```
+
+Default settings:
+
+- `predictor_type="hit"`
+- `model_name="Hierarchy-Transformers/HiT-MiniLM-L12-WordNetNoun"`
+- `search_method="random_anchors"`
+- `random_anchors_k=999`
+
+Required artifacts such as:
+- FAISS indices,
+- anchor vectors,
+- LightGBM models,
+- and reference percentile distributions
+
+are automatically downloaded and cached on first use.
+
+---
+
+## Important Compatibility Note
+
+The default configuration works out of the box and is the recommended setup.
+
+If you customize components such as:
+- the embedding model,
+- search method,
+- FAISS index,
+- anchor vectors,
+- or LightGBM model,
+
+you must ensure that all resources are compatible with each other.
+
+For example, a LightGBM model trained using:
+
+```python
+search_method="random_anchors"
+```
+
+should not be combined with:
+
+```python
+search_method="nearest_neighbor"
+```
+
+Similarly, FAISS indices, anchor vectors, percentile reference distributions, and LightGBM models must originate from the same embedding space and training configuration.
+
+Compatibility between custom resources is not validated automatically.
+
+---
+
+## Notebook Tutorial
+
+An interactive introduction is available in:
+
+```text
+notebooks/getting_started.ipynb
+```
+
+---
+
+## Repository Structure
+
+```text
+granuscore/
+├── src/
+│   └── granuscore/
+│       ├── pipeline.py
+│       ├── granularity_predictor.py
+│       ├── claim_splitter.py
+│       ├── bucket_output.py
+│       ├── cache.py
+│       └── artifacts.py
+├── notebooks/
+│   └── getting_started.ipynb
+├── training_scripts/
+├── evaluation/
+├── assets/
+├── data/ (needs to be externally downloaded)
+├── pyproject.toml
+├── LICENSE
+└── README.md
+```
+
+---
+
+## Reproducing Paper Experiments
+
+The datasets and precomputed resources required to reproduce the experiments from the paper are available here:
+
+https://drive.google.com/drive/folders/1mJdUENOxHEiuYn-_f1KRQ1PZggXJDnb4?usp=sharing
+
+Download the archive and extract it into the repository root:
+
+```bash
+unzip data.zip
+```
+
+This will create the expected directory structure used by the training and evaluation scripts.
+
+---
+
+## Training Pipeline
+
+Training uses precomputed `.pkl` feature files.
+
+1. Generate precomputed datasets:
+
+```text
+training_scripts/build_precalc_data/
+```
+
+2. Train LightGBM models:
+
+```bash
+python training_scripts/train_lgb_models.py
+```
+
+---
+
+## Citation
+Updated Citation information will be added after publication.
+
+```bibtex
+@misc{ellinger2026granuscore,
+  title={Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering},
+  author={Ellinger, Lukas and Fichtl, Alexander M. and Anschütz, Miriam and Groh, Georg},
+  year={2026}
+}
+```

granuscore-1.0.0/pyproject.toml

@@ -0,0 +1,66 @@
+[build-system]
+requires = ["setuptools>=69", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "granuscore"
+version = "1.0.0"
+description = "Granularity scoring for natural language"
+readme = "README.md"
+requires-python = ">=3.9,<3.13"
+
+authors = [
+  {name = "Lukas Ellinger", email = "lukas.ellinger@tum.de"},
+]
+
+license = "MIT"
+
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Intended Audience :: Science/Research",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Text Processing :: Linguistic",
+]
+
+keywords = [
+  "granularity",
+  "natural language processing",
+  "nlp",
+  "text analysis",
+  "semantic abstraction",
+  "specificity"
+]
+
+dependencies = [
+  "faiss-cpu>=1.13.2",
+  "hierarchy_transformers>=0.1.1",
+  "lightgbm>=4.6.0",
+  "spacy>=3.8.11",
+  "torch>=2.9.1",
+  "numpy>=1.26.4",
+  "platformdirs>=3.10.0",
+  "requests>=2.31.0",
+  "tqdm>=4.66.0",
+  "openai>=2.21.0"
+]
+
+[project.urls]
+Repository = "https://github.com/lukasellinger/granuscore"
+Issues = "https://github.com/lukasellinger/granuscore/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[project.optional-dependencies]
+dev = [
+  "optuna>=4.6.0",
+  "pygam>=0.12.0",
+  "python-dotenv>=1.2.1",
+  "ollama>=0.6.1",
+  "google-genai>=1.66.0",
+  "wget>=3.2"
+]

granuscore-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

granuscore-1.0.0/src/granuscore/__init__.py

@@ -0,0 +1,13 @@
+from .granularity_predictor import HitGranularityPredictor
+from .pipeline import GranuScore
+
+def granuscore(
+    texts,
+    **kwargs,
+):
+    return GranuScore()(
+        texts,
+        **kwargs,
+    )
+
+__all__ = ["GranuScore", "granuscore", "HitGranularityPredictor"]

granuscore-1.0.0/src/granuscore/artifcats.py

@@ -0,0 +1,116 @@
+import hashlib
+import os
+import shutil
+from dataclasses import dataclass
+from pathlib import Path
+
+import requests
+from importlib.resources import files
+from platformdirs import user_cache_dir
+
+
+def _sha256(path: Path) -> str:
+    h = hashlib.sha256()
+    with path.open("rb") as f:
+        for chunk in iter(lambda: f.read(1024 * 1024), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+@dataclass(frozen=True)
+class ArtifactSpec:
+    name: str
+    subdir: str              # "faiss" or "models"
+    url: str | None = None   # if None: must be bundled or already cached
+    sha256: str | None = None
+
+
+class ArtifactManager:
+    """
+    Resolution order:
+      1) cache (and sha256 if provided)
+      2) bundled assets inside the package -> copied to cache
+      3) download from spec.url (public) -> cached
+    """
+
+    def __init__(self, app_name: str = "granuscore"):
+        self.cache_root = Path(user_cache_dir(app_name))
+        self.cache_root.mkdir(parents=True, exist_ok=True)
+
+    def ensure(self, spec: ArtifactSpec, force: bool = False) -> Path:
+        dst_dir = self.cache_root / spec.subdir
+        dst_dir.mkdir(parents=True, exist_ok=True)
+        dst = dst_dir / spec.name
+
+        # 1) cache hit
+        if dst.exists() and not force:
+            if spec.sha256 is None or _sha256(dst) == spec.sha256:
+                return dst
+            dst.unlink(missing_ok=True)
+
+        # 2) bundled asset
+        bundled = files("granuscore") / "../../assets" / spec.subdir / spec.name
+        if bundled.is_file() and not force:
+            self._copy_to_cache(bundled, dst)
+            self._verify_hash_if_needed(dst, spec)
+            return dst
+
+        # 3) public download
+        if not spec.url:
+            raise FileNotFoundError(
+                f"{spec.name} not found in cache and not bundled, and no public download URL provided."
+            )
+
+        self._download_public(spec.url, dst)
+        self._verify_hash_if_needed(dst, spec)
+        return dst
+
+    def _verify_hash_if_needed(self, path: Path, spec: ArtifactSpec) -> None:
+        if spec.sha256 is None:
+            return
+        got = _sha256(path)
+        if got != spec.sha256:
+            path.unlink(missing_ok=True)
+            raise ValueError(
+                f"Hash mismatch for {spec.name}. Expected {spec.sha256}, got {got}."
+            )
+
+    def _copy_to_cache(self, src_path: Path, dst: Path) -> None:
+        tmp = dst.with_suffix(dst.suffix + ".part")
+        with src_path.open("rb") as src, tmp.open("wb") as out:
+            shutil.copyfileobj(src, out)
+        tmp.replace(dst)
+
+    def _download_public(self, url: str, dst: Path) -> None:
+        tmp = dst.with_suffix(dst.suffix + ".part")
+
+        with requests.get(url, stream=True, timeout=60, allow_redirects=True) as r:
+            # Common mistake: GitHub "blob" links download HTML
+            if r.status_code >= 400 and "github.com/" in url and "/blob/" in url:
+                raise requests.HTTPError(
+                    "You are using a GitHub 'blob' URL (HTML). Use a release asset URL "
+                    "(.../releases/download/...) or a raw.githubusercontent.com URL."
+                )
+
+            r.raise_for_status()
+            total = int(r.headers.get("Content-Length", 0)) or None
+
+            try:
+                from tqdm import tqdm  # type: ignore
+                bar = tqdm(total=total, unit="B", unit_scale=True, desc=f"Downloading {dst.name}")
+            except Exception:
+                bar = None
+
+            try:
+                with tmp.open("wb") as f:
+                    for chunk in r.iter_content(chunk_size=1024 * 1024):
+                        if not chunk:
+                            continue
+                        f.write(chunk)
+                        if bar is not None:
+                            bar.update(len(chunk))
+            finally:
+                if bar is not None:
+                    bar.close()
+
+        os.replace(tmp, dst)