datapruning 2.1.2__tar.gz → 2.1.4__tar.gz

datapruning-2.1.4/PKG-INFO

@@ -0,0 +1,150 @@
+Metadata-Version: 2.4
+Name: datapruning
+Version: 2.1.4
+Summary: Intelligent dataset optimization for cleaner, smaller training data
+License: Proprietary
+Project-URL: Homepage, https://www.datapruning.com
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: scikit-learn>=1.3.0
+Provides-Extra: torch
+Requires-Dist: torch>=2.0.0; extra == "torch"
+Provides-Extra: clip
+Requires-Dist: torch>=2.0.0; extra == "clip"
+Requires-Dist: transformers>=4.30.0; extra == "clip"
+Requires-Dist: pillow>=9.0.0; extra == "clip"
+Dynamic: license-file
+
+# DataPruning
+
+Intelligent dataset optimization — reduces dataset size by selecting the most informative rows while preserving model quality.
+
+## Installation
+
+```bash
+pip install datapruning
+```
+
+For image/text multimodal support (CLIP encoding):
+
+```bash
+pip install datapruning[clip]
+```
+
+## Quick Start
+
+### CSV / Tabular Data
+
+```python
+import pandas as pd
+from datapruning.sdk import DatasetEngine
+
+df = pd.read_csv("dataset.csv")
+engine = DatasetEngine(df, target_col="label")
+
+# Analyze the dataset
+analysis = engine.analyze()
+
+# Get a recommendation
+rec = engine.recommend()
+print(f"Best method: {rec.display_name} ({rec.confidence}% confidence)")
+
+# Optimize — keep 50% of most informative rows
+result = engine.optimize(keep_ratio=0.5)
+print(f"Kept {result['runtime_seconds']:.1f}s — {result['explainability']['rows_removed']} rows removed")
+
+# Download the optimized dataset
+result["optimized_df"].to_csv("optimized.csv", index=False)
+```
+
+### Image Datasets
+
+```python
+from datapruning.sdk import DatasetEngine
+
+# Create engine from a folder of images
+engine = DatasetEngine.from_images(
+    folder_path="./photos",
+    labels=[0, 1, 0, 1, ...],  # one label per image
+    keep_ratio=0.5,
+)
+
+# Analyze and optimize
+analysis = engine.analyze()
+result = engine.optimize(keep_ratio=0.5)
+result["optimized_df"].to_csv("optimized_images.csv", index=False)
+```
+
+### Text Datasets
+
+```python
+engine = DatasetEngine.from_texts(
+    texts=["label A sample", "label B sample", ...],
+    labels=[0, 1, 0, 1, ...],
+)
+result = engine.optimize(keep_ratio=0.5)
+```
+
+## Algorithms
+
+DataPruning selects the best algorithm automatically based on dataset characteristics:
+
+| Method | What it does | Best for |
+|--------|-------------|----------|
+| **Margin Optimization** | Analyzes classification confidence gap via cross-validation, keeps boundary samples | General tabular data |
+| **Importance Scoring** | Measures sample difficulty via model prediction confidence across CV folds | Quick baseline, any dataset |
+| **Diversity Selection** | Picks the most representative subset (K-Center) | High-dimensional data |
+| **Noise Filtering** | Drops redundant and potentially mislabeled samples via CV-based difficulty | Noisy/messy data |
+| **Full Pipeline** | Combines all signals for maximum quality | Complex datasets |
+
+All scoring methods use **out-of-fold (OOF) model logits** — each sample is scored by a model that never saw it during training, producing unbiased difficulty estimates. The optimizer also includes a **"must beat random" gate** that validates the selected strategy actually improves over random subsampling on an internal holdout; if not, it safely falls back to random selection.
+
+## Limits
+
+- Minimum: 1,000 rows (tabular) or 10 images
+- Maximum: 300,000 rows (tabular) or 1,000 images
+- Supported image formats: JPG, PNG, BMP, GIF, TIFF, WebP
+
+## Requirements
+
+- Python >= 3.10
+- Pandas 2.0+
+- NumPy 1.24+
+- scikit-learn 1.3+
+
+Optional (for image/text support):
+- PyTorch 2.0+
+- transformers 4.30+
+- Pillow 9.0+
+
+## Features
+
+- Runs locally — no data uploaded anywhere
+- Auto-detects task type (classification/regression) and target column
+- Auto-encodes categorical/string columns
+- 5 optimization algorithms with automatic selection
+- Preserves minority class balance
+- CSV and image dataset support
+- Pure Python — works on any platform
+
+## Links
+
+- Website: [datapruning.com](https://www.datapruning.com)
+- PyPI: [pypi.org/project/datapruning](https://pypi.org/project/datapruning/)
+
+## License
+
+Proprietary. See LICENSE file.

datapruning-2.1.4/README.md

@@ -0,0 +1,120 @@
+# DataPruning
+
+Intelligent dataset optimization — reduces dataset size by selecting the most informative rows while preserving model quality.
+
+## Installation
+
+```bash
+pip install datapruning
+```
+
+For image/text multimodal support (CLIP encoding):
+
+```bash
+pip install datapruning[clip]
+```
+
+## Quick Start
+
+### CSV / Tabular Data
+
+```python
+import pandas as pd
+from datapruning.sdk import DatasetEngine
+
+df = pd.read_csv("dataset.csv")
+engine = DatasetEngine(df, target_col="label")
+
+# Analyze the dataset
+analysis = engine.analyze()
+
+# Get a recommendation
+rec = engine.recommend()
+print(f"Best method: {rec.display_name} ({rec.confidence}% confidence)")
+
+# Optimize — keep 50% of most informative rows
+result = engine.optimize(keep_ratio=0.5)
+print(f"Kept {result['runtime_seconds']:.1f}s — {result['explainability']['rows_removed']} rows removed")
+
+# Download the optimized dataset
+result["optimized_df"].to_csv("optimized.csv", index=False)
+```
+
+### Image Datasets
+
+```python
+from datapruning.sdk import DatasetEngine
+
+# Create engine from a folder of images
+engine = DatasetEngine.from_images(
+    folder_path="./photos",
+    labels=[0, 1, 0, 1, ...],  # one label per image
+    keep_ratio=0.5,
+)
+
+# Analyze and optimize
+analysis = engine.analyze()
+result = engine.optimize(keep_ratio=0.5)
+result["optimized_df"].to_csv("optimized_images.csv", index=False)
+```
+
+### Text Datasets
+
+```python
+engine = DatasetEngine.from_texts(
+    texts=["label A sample", "label B sample", ...],
+    labels=[0, 1, 0, 1, ...],
+)
+result = engine.optimize(keep_ratio=0.5)
+```
+
+## Algorithms
+
+DataPruning selects the best algorithm automatically based on dataset characteristics:
+
+| Method | What it does | Best for |
+|--------|-------------|----------|
+| **Margin Optimization** | Analyzes classification confidence gap via cross-validation, keeps boundary samples | General tabular data |
+| **Importance Scoring** | Measures sample difficulty via model prediction confidence across CV folds | Quick baseline, any dataset |
+| **Diversity Selection** | Picks the most representative subset (K-Center) | High-dimensional data |
+| **Noise Filtering** | Drops redundant and potentially mislabeled samples via CV-based difficulty | Noisy/messy data |
+| **Full Pipeline** | Combines all signals for maximum quality | Complex datasets |
+
+All scoring methods use **out-of-fold (OOF) model logits** — each sample is scored by a model that never saw it during training, producing unbiased difficulty estimates. The optimizer also includes a **"must beat random" gate** that validates the selected strategy actually improves over random subsampling on an internal holdout; if not, it safely falls back to random selection.
+
+## Limits
+
+- Minimum: 1,000 rows (tabular) or 10 images
+- Maximum: 300,000 rows (tabular) or 1,000 images
+- Supported image formats: JPG, PNG, BMP, GIF, TIFF, WebP
+
+## Requirements
+
+- Python >= 3.10
+- Pandas 2.0+
+- NumPy 1.24+
+- scikit-learn 1.3+
+
+Optional (for image/text support):
+- PyTorch 2.0+
+- transformers 4.30+
+- Pillow 9.0+
+
+## Features
+
+- Runs locally — no data uploaded anywhere
+- Auto-detects task type (classification/regression) and target column
+- Auto-encodes categorical/string columns
+- 5 optimization algorithms with automatic selection
+- Preserves minority class balance
+- CSV and image dataset support
+- Pure Python — works on any platform
+
+## Links
+
+- Website: [datapruning.com](https://www.datapruning.com)
+- PyPI: [pypi.org/project/datapruning](https://pypi.org/project/datapruning/)
+
+## License
+
+Proprietary. See LICENSE file.

{datapruning-2.1.2 → datapruning-2.1.4}/datapruning/__init__.py

@@ -5,6 +5,8 @@ import pandas as pd
 from . import pipeline
 from .sdk import DatasetEngine
 
+__all__ = ["DatasetEngine", "Pruner", "prune_dataframe"]
+
 logger = logging.getLogger(__name__)
 
 MIN_ROWS = 1000

{datapruning-2.1.2 → datapruning-2.1.4}/datapruning/algorithms/base.py

@@ -2,6 +2,7 @@ from __future__ import annotations
 from abc import ABC, abstractmethod
 from dataclasses import dataclass, field
 from typing import Optional
+import numpy as np
 import pandas as pd
 
 
@@ -15,6 +16,8 @@ class OptimizationResult:
     minority_count_before: Optional[int]
     minority_count_after: Optional[int]
     runtime_seconds: float
+    beneficial: bool = True
+    strategy_method: str = ""
 
 
 @dataclass
@@ -27,6 +30,7 @@ class Explanation:
     estimated_training_gain_pct: float
     memory_saved_pct: float
     limitations: list[str] = field(default_factory=list)
+    beneficial: bool = True
 
 
 class BaseOptimizer(ABC):
@@ -45,18 +49,77 @@ class BaseOptimizer(ABC):
         keep_idx = self._select_rows(X, y, keep_ratio)
         return X.loc[keep_idx], y.loc[keep_idx]
 
+    def _beat_random_gate(
+        self,
+        X: pd.DataFrame,
+        y: pd.Series,
+        keep_ratio: float,
+        n_random: int = 3,
+        test_size: float = 0.3,
+        seed: int = 0,
+    ) -> bool:
+        """Check whether this strategy beats random on an internal holdout.
+
+        Splits the data 70/30, trains the strategy on the 70% subset,
+        validates on the 30% holdout, and compares with random subsets.
+        Returns True if the strategy is beneficial.
+        """
+        from sklearn.model_selection import train_test_split
+        from sklearn.linear_model import LogisticRegression
+
+        if len(y) < 50:
+            return True
+        y_vals = y.values if hasattr(y, 'values') else np.asarray(y)
+        n_unique = len(np.unique(y_vals))
+        if n_unique < 2 or n_unique >= len(y_vals) * 0.8:
+            return True
+
+        tr_idx, va_idx = train_test_split(
+            np.arange(len(y)), test_size=test_size,
+            random_state=seed, stratify=y.values,
+        )
+        X_tr, y_tr = X.iloc[tr_idx], y.iloc[tr_idx]
+        X_va, y_va = X.iloc[va_idx], y.iloc[va_idx]
+        n_keep = max(1, int(len(tr_idx) * keep_ratio))
+
+        try:
+            self.fit(X_tr, y_tr)
+            s_keep = self._select_rows(X_tr, y_tr, keep_ratio)
+            m = LogisticRegression(max_iter=300, C=0.1, random_state=seed)
+            m.fit(X_tr.loc[s_keep], y_tr.loc[s_keep])
+            strat_acc = m.score(X_va, y_va)
+
+            rng = np.random.RandomState(seed)
+            random_accs = []
+            for s in range(n_random):
+                r_idx = rng.choice(len(tr_idx), n_keep, replace=False)
+                r_keep = X_tr.index[r_idx]
+                m2 = LogisticRegression(max_iter=300, C=0.1, random_state=seed)
+                m2.fit(X_tr.loc[r_keep], y_tr.loc[r_keep])
+                random_accs.append(m2.score(X_va, y_va))
+
+            return strat_acc >= np.mean(random_accs)
+        except Exception:
+            return True
+
+    def _random_select(self, X: pd.DataFrame, y: pd.Series, keep_ratio: float, seed: int = 42) -> pd.Index:
+        n_target = max(1, int(len(y) * keep_ratio))
+        rng = np.random.RandomState(seed)
+        indices = rng.choice(len(y), n_target, replace=False)
+        return X.index[indices]
+
     def optimize(self, df: pd.DataFrame, target_col: str, keep_ratio: float = 0.5) -> OptimizationResult:
         import time
-        import numpy as np
-        from sklearn.preprocessing import LabelEncoder
 
         X, y = df.drop(columns=[target_col]), df[target_col].copy()
 
         for col in X.columns:
             if not np.issubdtype(X[col].dtype, np.number):
+                from sklearn.preprocessing import LabelEncoder
                 X[col] = LabelEncoder().fit_transform(X[col].astype(str))
 
         if not np.issubdtype(y.dtype, np.number):
+            from sklearn.preprocessing import LabelEncoder
             y = pd.Series(LabelEncoder().fit_transform(y.astype(str)), index=y.index)
 
         X = X.fillna(0).astype(np.float64)
@@ -65,6 +128,15 @@ class BaseOptimizer(ABC):
         self.fit(X, y)
         X_res, y_res = self.transform(X, y, keep_ratio)
         runtime = time.time() - t0
+
+        beneficial = self._beat_random_gate(X, y, keep_ratio)
+        if not beneficial:
+            keep_idx = self._random_select(X, y, keep_ratio)
+            X_res, y_res = X.loc[keep_idx], y.loc[keep_idx]
+            strategy_method = "random (fallback)"
+        else:
+            strategy_method = self.display_name
+
         minority_after = int(y_res.value_counts().min()) if y_res.nunique() > 1 else None
         optimized_df = X_res.copy()
         optimized_df[target_col] = y_res.values
@@ -77,6 +149,8 @@ class BaseOptimizer(ABC):
             minority_count_before=minority_before,
             minority_count_after=minority_after,
             runtime_seconds=runtime,
+            beneficial=beneficial,
+            strategy_method=strategy_method,
         )
 
     def explain(self, result: OptimizationResult) -> Explanation:
@@ -88,7 +162,7 @@ class BaseOptimizer(ABC):
                 (result.minority_count_after or 0) / result.minority_count_before * 100, 1
             )
         return Explanation(
-            algorithm=self.display_name,
+            algorithm=result.strategy_method or self.display_name,
             why_selected=self.why_selected(),
             rows_removed=rows_removed,
             reduction_pct=reduction_pct,
@@ -96,6 +170,7 @@ class BaseOptimizer(ABC):
             estimated_training_gain_pct=reduction_pct,
             memory_saved_pct=reduction_pct,
             limitations=self.limitations(),
+            beneficial=result.beneficial,
         )
 
     @staticmethod

datapruning-2.1.4/datapruning/algorithms/el2n_scorer.py

@@ -0,0 +1,79 @@
+from __future__ import annotations
+import numpy as np
+import pandas as pd
+from datapruning.algorithms.oof_utils import compute_oof_logits, logits_to_el2n
+from datapruning.algorithms.base import BaseOptimizer
+from datapruning.algorithms.registry import register
+
+
+@register
+class EL2NScorerOptimizer(BaseOptimizer):
+    key = "el2n"
+    display_name = "Importance Scoring"
+
+    def __init__(self):
+        self._scores = None
+
+    def fit(self, X: pd.DataFrame, y: pd.Series) -> "EL2NScorerOptimizer":
+        X_arr = X.values.astype(np.float64)
+        y_arr = y.values.astype(int)
+        classes = np.unique(y_arr)
+        if len(classes) < 2:
+            self._scores = pd.Series(np.zeros(len(y_arr)), index=y.index)
+            return self
+        try:
+            logits = compute_oof_logits(X_arr, y_arr)
+            el2n = logits_to_el2n(logits, y_arr)
+            self._scores = pd.Series(el2n, index=y.index)
+        except Exception:
+            centroids = np.zeros((len(classes), X_arr.shape[1]))
+            for i, c in enumerate(classes):
+                centroids[i] = X_arr[y_arr == c].mean(axis=0)
+            scores = np.zeros(len(y_arr), dtype=np.float64)
+            for i, c in enumerate(classes):
+                dists = np.linalg.norm(X_arr - centroids[i], axis=1)
+                scores = np.maximum(scores, dists)
+            self._scores = pd.Series(scores, index=y.index)
+        return self
+
+    def _select_rows(self, X: pd.DataFrame, y: pd.Series, keep_ratio: float) -> pd.Index:
+        if self._scores is None:
+            self.fit(X, y)
+        classes = y.unique()
+        if len(classes) < 2:
+            n_target = max(1, int(len(y) * keep_ratio))
+            return y.sample(n=n_target, random_state=42).index
+        minority_class = y.value_counts().idxmin()
+        minority_idx = y[y == minority_class].index
+        majority_idx = y[y != minority_class].index
+        n_total_target = max(1, int(len(y) * keep_ratio))
+        n_minority = min(len(minority_idx), int(n_total_target * 0.8))
+        n_majority_target = max(n_total_target - n_minority, 0)
+        majority_scores = self._scores.loc[majority_idx].sort_values(ascending=False)
+        kept_majority_idx = majority_scores.index[:n_majority_target]
+        kept_minority_idx = minority_idx[:n_minority]
+        return kept_minority_idx.append(kept_majority_idx)
+
+    def why_selected(self) -> str:
+        return ("Measures how difficult each sample is for the model to classify by analyzing "
+                "model prediction confidence across cross-validation folds. Keeps the most "
+                "informative boundary samples that help the model learn better decision regions. "
+                "Works reliably on any dataset.")
+
+    def limitations(self) -> list[str]:
+        return [
+            "Requires a quick cross-validation step during analysis.",
+            "May select outliers on very noisy datasets.",
+        ]
+
+    @staticmethod
+    def fit_score(analysis: dict) -> float:
+        scan = analysis.get("scan", analysis)
+        task_type = scan.get("task_type", "unknown")
+        n_rows = scan.get("rows", 0)
+
+        if task_type == "regression":
+            return 0.0
+        if n_rows < 200:
+            return 0.2
+        return 0.4

{datapruning-2.1.2 → datapruning-2.1.4}/datapruning/algorithms/kcenter_select.py

@@ -48,9 +48,11 @@ class KCenterSelectOptimizer(BaseOptimizer):
         n_cols = scan.get("columns", 0)
 
         if task_type == "regression":
-            return 0.2
+            return 0.3
+        if n_rows < 500:
+            return 0.6
         if n_cols > n_rows:
-            return 0.8
+            return 0.9
         if n_cols > 50:
-            return 0.65
-        return 0.4
+            return 0.85
+        return 0.8

{datapruning-2.1.2 → datapruning-2.1.4}/datapruning/algorithms/margin_scorer.py

@@ -1,7 +1,7 @@
 from __future__ import annotations
 import numpy as np
 import pandas as pd
-from sklearn.linear_model import LogisticRegression
+from datapruning.algorithms.oof_utils import compute_oof_logits, logits_to_margin
 from datapruning.algorithms.base import BaseOptimizer
 from datapruning.algorithms.registry import register
 
@@ -15,18 +15,16 @@ class MarginScorerOptimizer(BaseOptimizer):
         self._scores = None
 
     def fit(self, X: pd.DataFrame, y: pd.Series) -> "MarginScorerOptimizer":
-        X_arr, y_arr = X.values.astype(np.float64), y.values
+        X_arr = X.values.astype(np.float64)
+        y_arr = y.values.astype(int)
         classes = np.unique(y_arr)
         if len(classes) < 2:
             self._scores = pd.Series(np.zeros(len(y_arr)), index=y.index)
             return self
         try:
-            lr = LogisticRegression(max_iter=1000, random_state=42)
-            lr.fit(X_arr, y_arr)
-            probs = lr.predict_proba(X_arr)
-            class_to_col = {c: i for i, c in enumerate(lr.classes_)}
-            true_probs = np.array([probs[i, class_to_col[y_arr[i]]] for i in range(len(y_arr))])
-            self._scores = pd.Series(true_probs, index=y.index)
+            logits = compute_oof_logits(X_arr, y_arr)
+            margin = logits_to_margin(logits, y_arr)
+            self._scores = pd.Series(margin, index=y.index)
         except Exception:
             feat_norms = np.linalg.norm(X_arr, axis=1)
             self._scores = pd.Series(feat_norms, index=y.index)
@@ -37,25 +35,29 @@ class MarginScorerOptimizer(BaseOptimizer):
             self.fit(X, y)
         classes = y.unique()
         if len(classes) < 2:
-            n_target = int(len(y) * keep_ratio)
-            return y.nlargest(n_target).index
+            n_target = max(1, int(len(y) * keep_ratio))
+            return y.sample(n=n_target, random_state=42).index
         minority_class = y.value_counts().idxmin()
         minority_idx = y[y == minority_class].index
         majority_idx = y[y != minority_class].index
-        n_total_target = int(len(y) * keep_ratio)
-        n_majority_target = max(n_total_target - len(minority_idx), 0)
-        majority_scores = self._scores.loc[majority_idx].sort_values(ascending=False)
+        n_total_target = max(1, int(len(y) * keep_ratio))
+        n_minority = min(len(minority_idx), int(n_total_target * 0.8))
+        n_majority_target = max(n_total_target - n_minority, 0)
+        majority_scores = self._scores.loc[majority_idx].sort_values(ascending=True)
         kept_majority_idx = majority_scores.index[:n_majority_target]
-        return minority_idx.append(kept_majority_idx)
+        kept_minority_idx = minority_idx[:n_minority]
+        return kept_minority_idx.append(kept_majority_idx)
 
     def why_selected(self) -> str:
-        return ("Analyzes how confidently each data point is classified, keeping the most "
-                "informative samples near decision boundaries. Automatically preserves all "
-                "minority-class examples to maintain balanced training data.")
+        return ("Analyzes how confidently each data point is classified by looking at "
+                "the gap between the top two predicted classes across cross-validation "
+                "folds. Keeps the most informative samples near decision boundaries. "
+                "Automatically preserves all minority-class examples to maintain "
+                "balanced training data.")
 
     def limitations(self) -> list[str]:
         return [
-            "Requires a quick model fitting step during analysis.",
+            "Requires a quick cross-validation step during analysis.",
             "May be less effective on extremely high-dimensional datasets.",
         ]
 
@@ -64,14 +66,9 @@ class MarginScorerOptimizer(BaseOptimizer):
         scan = analysis.get("scan", analysis)
         task_type = scan.get("task_type", "unknown")
         n_rows = scan.get("rows", 0)
-        n_cols = scan.get("columns", 0)
 
         if task_type == "regression":
             return 0.0
         if n_rows < 500:
-            return 0.1
-        if n_rows > 200_000:
-            return 0.6
-        if n_cols > 100:
-            return 0.7
-        return 0.85
+            return 0.2
+        return 0.3

datapruning-2.1.4/datapruning/algorithms/oof_utils.py

@@ -0,0 +1,75 @@
+from __future__ import annotations
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import StratifiedKFold
+
+
+def compute_oof_logits(
+    X: np.ndarray,
+    y: np.ndarray,
+    n_folds: int = 5,
+    seed: int = 0,
+    C: float = 0.1,
+    max_iter: int = 300,
+) -> np.ndarray:
+    """Compute out-of-fold log-probabilities via stratified K-fold CV.
+
+    Each sample is scored by a model that never saw it during training,
+    producing unbiased per-sample logits.  This is the correct input for
+    EL2N, margin, and sliding-window difficulty scores.
+
+    Returns
+    -------
+    logits : ndarray of shape (n_samples, n_classes)
+        Log-probability of each class for every sample.
+    """
+    X = np.asarray(X, dtype=np.float64)
+    y = np.asarray(y).astype(int)
+    n_classes = len(np.unique(y))
+    out = np.zeros((len(X), n_classes), dtype=np.float64)
+
+    skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=seed)
+    for tr_idx, va_idx in skf.split(X, y):
+        model = LogisticRegression(
+            max_iter=max_iter, C=C, random_state=seed, solver="lbfgs"
+        )
+        model.fit(X[tr_idx], y[tr_idx])
+        log_proba = model.predict_log_proba(X[va_idx])
+        class_cols = model.classes_.astype(int)
+        out[np.ix_(va_idx, class_cols)] = log_proba
+
+    return out
+
+
+def logits_to_el2n(logits: np.ndarray, y: np.ndarray) -> np.ndarray:
+    """EL2N score from model logits: ||softmax(z) - onehot(y)||.
+
+    Higher values indicate harder / more informative samples.
+    """
+    z = logits - logits.max(axis=1, keepdims=True)
+    p = np.exp(z)
+    p /= p.sum(axis=1, keepdims=True)
+    oh = np.zeros_like(p)
+    oh[np.arange(len(y)), y] = 1.0
+    return np.linalg.norm(p - oh, axis=1)
+
+
+def logits_to_margin(logits: np.ndarray, y: np.ndarray) -> np.ndarray:
+    """Margin score from model logits: confidence gap between top-1 and top-2.
+
+    Lower values indicate harder / more ambiguous samples (near decision
+    boundary).  The score is returned as (1 - margin) so that higher
+    values consistently mean "harder" across all scorers.
+    """
+    z = logits - logits.max(axis=1, keepdims=True)
+    p = np.exp(z)
+    p /= p.sum(axis=1, keepdims=True)
+    n = len(y)
+    margins = np.zeros(n, dtype=np.float64)
+    for i in range(n):
+        sorted_p = np.sort(p[i])[::-1]
+        if len(sorted_p) >= 2:
+            margins[i] = sorted_p[0] - sorted_p[1]
+        else:
+            margins[i] = sorted_p[0]
+    return 1.0 - margins