resolvekit 0.0.1__py3-none-any.whl

resolvekit/calibration/README.md

@@ -0,0 +1,351 @@
+# Calibration Module
+
+## Purpose
+
+The calibration module converts raw matcher scores into calibrated confidence probabilities that reflect actual precision, enabling reliable threshold-based decision making.
+
+## Components
+
+### Core Components
+
+**Currently Implemented:**
+
+1. **CalibrationModel** (`models.py`)
+   - Pydantic schema for calibration model configuration
+   - JSON loading and validation
+   - Model metadata (ECE, training date, notes)
+
+2. **Feature Extractor** (`features.py`)
+   - Extracts standardized features from candidates
+   - Converts matcher-specific features to normalized values
+   - Handles missing features with sentinel values
+   - Optional numpy vectorization for batch processing
+
+3. **Calibrator** (`calibrator.py`)
+   - Applies calibration models to candidates
+   - Returns probability P(correct match | features)
+   - Supports logistic regression models
+   - Heuristic fallback when no model available
+   - Batch processing with optional numpy acceleration
+
+**Planned for Future Phases:**
+
+4. **Model Trainer** (`trainer.py` - not yet implemented)
+   - Trains calibration models on labeled data
+   - Evaluates calibration quality (ECE - Expected Calibration Error)
+   - Exports models to JSON for data packs
+
+5. **Score Fusion** (`fusion.py` - not yet implemented)
+   - Combines scores from multiple matchers
+   - Handles missing features (when matchers not invoked)
+   - Normalizes scores to comparable scales
+
+## Feature Set
+
+Features extracted per (query, candidate) pair:
+
+### Match Type Features (binary)
+- `f_exact_code`: Matched via exact code lookup
+- `f_canonical_exact`: Matched via canonical name
+- `f_alias_exact`: Matched via exact alias
+- `f_alias_type_*`: One-hot encoding of alias type (canonical/endonym/exonym/abbr/code)
+
+### Similarity Features (continuous)
+- `f_fts_score`: BM25-like score from FTS
+- `f_fts_rank_inv`: Inverse rank (1/rank)
+- `f_edit_distance_norm`: Normalized edit distance
+- `f_trigram_jaccard`: Trigram Jaccard similarity
+
+### Constraint Features (binary)
+- `f_parent_valid`: Passes parent constraint
+- `f_type_valid`: Passes type constraint
+- `f_date_valid`: Passes temporal constraint
+
+### Semantic Features (optional)
+- `f_sem_used`: Semantic matcher was invoked
+- `f_sem_sim`: Cosine similarity from embeddings (0-1)
+
+### Context Features
+- `f_ambiguity_flag`: Query in ambiguity registry
+- `f_region_hint_match`: Candidate matches region hint
+
+## Calibration Model
+
+### Logistic Regression (default)
+
+```python
+import numpy as np
+
+def calibrate_logistic(features: np.ndarray, weights: np.ndarray, bias: float) -> float:
+    """Compute calibrated probability via logistic regression."""
+    logit = np.dot(weights, features) + bias
+    probability = 1 / (1 + np.exp(-logit))
+    return probability
+```
+
+### Model Format (JSON)
+
+```json
+{
+  "type": "logistic",
+  "features": ["f_exact_code", "f_canonical_exact", ...],
+  "weights": [2.3, 1.6, 0.9, ...],
+  "bias": -1.4,
+  "ece": 0.032,
+  "trained_on": "2025-10-15",
+  "notes": "Global calibration; overlays may append."
+}
+```
+
+## Calibration Quality Metrics
+
+- **ECE (Expected Calibration Error)**: Measures deviation between predicted probabilities and observed accuracy
+- **Brier Score**: Mean squared error of probabilistic predictions
+- **Reliability Diagram**: Plots predicted vs observed probabilities in bins
+
+Target: ECE < 0.05 (well-calibrated)
+
+## Design Principles
+
+1. **Interpretable**: Use simple models (logistic) for explainability
+2. **Calibrated**: Probabilities should match empirical accuracy
+3. **Versioned**: Models versioned with data packs
+4. **Updatable**: Overlays can fine-tune calibration with delta weights
+
+## Usage Examples
+
+### Basic Usage - Heuristic Fallback
+
+When no calibration model is available, the calibrator uses rule-based heuristics:
+
+```python
+from resolvekit.calibration import Calibrator
+from resolvekit.types import Candidate, Entity, EntityType, MatcherType
+
+# Create calibrator without model (uses heuristic)
+calibrator = Calibrator(model=None)
+
+# Example candidate from exact code matcher
+candidate = Candidate(
+    entity=Entity(
+        dcid="country/USA",
+        canonical_name="United States",
+        entity_type=EntityType.COUNTRY,
+    ),
+    score=1.0,
+    matcher_type=MatcherType.EXACT_CODE,
+    features={"exact_code": True, "code_system": "iso3"},
+)
+
+# Calibrate to get confidence probability
+confidence = calibrator.calibrate(candidate)
+print(f"Confidence: {confidence}")  # 0.95 for exact code matches
+```
+
+### Loading and Using a Trained Model
+
+Load a calibration model from a data pack and use it for calibration:
+
+```python
+from pathlib import Path
+from resolvekit.calibration import Calibrator, load_calibration_model
+
+# Load model from JSON file in data pack
+model_path = Path("data/base/calibration_model.json")
+model = load_calibration_model(model_path)
+
+if model:
+    print(f"Loaded model with {len(model.features)} features")
+    print(f"ECE: {model.ece}")  # Expected Calibration Error
+
+    # Create calibrator with model
+    calibrator = Calibrator(model=model)
+
+    # Calibrate candidates using logistic regression
+    confidence = calibrator.calibrate(candidate)
+    print(f"Model-calibrated confidence: {confidence}")
+else:
+    print("Model not found, falling back to heuristic")
+    calibrator = Calibrator(model=None)
+```
+
+### Batch Processing for Efficiency
+
+Process multiple candidates efficiently using batch calibration:
+
+```python
+from resolvekit.calibration import Calibrator
+
+# Prepare multiple candidates
+candidates = [
+    Candidate(
+        entity=Entity(dcid="country/USA", canonical_name="United States",
+                     entity_type=EntityType.COUNTRY),
+        score=1.0,
+        matcher_type=MatcherType.EXACT_CODE,
+        features={"exact_code": True},
+    ),
+    Candidate(
+        entity=Entity(dcid="country/FRA", canonical_name="France",
+                     entity_type=EntityType.COUNTRY),
+        score=0.9,
+        matcher_type=MatcherType.CANONICAL_NAME,
+        features={"canonical_exact": True},
+    ),
+    Candidate(
+        entity=Entity(dcid="country/DEU", canonical_name="Germany",
+                     entity_type=EntityType.COUNTRY),
+        score=0.75,
+        matcher_type=MatcherType.FUZZY,
+        features={"fuzzy_score": 0.78, "fts_score": 0.82},
+    ),
+]
+
+calibrator = Calibrator(model=None)
+
+# Batch calibration (uses numpy vectorization if available)
+confidences = calibrator.calibrate_batch(candidates)
+
+# Results correspond to input order
+for candidate, confidence in zip(candidates, confidences):
+    print(f"{candidate.entity.canonical_name}: {confidence:.2f}")
+# Output:
+# United States: 0.95
+# France: 0.90
+# Germany: 0.73
+```
+
+### Feature Extraction
+
+Extract standardized features from candidates for analysis or custom calibration:
+
+```python
+from resolvekit.calibration import FeatureExtractor
+
+extractor = FeatureExtractor()
+
+# Extract features from a candidate
+features = extractor.extract(candidate)
+
+# Features dict contains standardized feature values
+print(features)
+# {
+#   'f_exact_code': 1.0,
+#   'f_canonical_exact': 0.0,
+#   'f_alias_exact': 0.0,
+#   'f_fts_score': -1.0,  # Sentinel value (not used)
+#   'f_fts_rank_inv': -1.0,
+#   'f_edit_similarity': -1.0,
+#   'f_trigram_jaccard': -1.0,
+#   'f_fuzzy_score': -1.0,
+#   'f_parent_valid': 0.0,
+#   'f_parent_depth': 0.0,
+#   'f_type_valid': 0.0,
+#   'f_date_valid': 0.0,
+#   'f_membership_valid': 0.0,
+# }
+
+# Extract features from multiple candidates
+feature_dicts = extractor.extract_batch(candidates)
+# Returns list of dicts (or numpy array if numpy available)
+```
+
+### Creating a Calibration Model
+
+Define a custom calibration model programmatically:
+
+```python
+from resolvekit.calibration import CalibrationModel
+from datetime import date
+
+# Create a simple logistic regression model
+model = CalibrationModel(
+    type="logistic",
+    features=["f_exact_code", "f_canonical_exact", "f_fts_score"],
+    weights=[4.0, 3.5, 1.5],
+    bias=-2.0,
+    ece=0.025,
+    trained_on=date(2025, 10, 15),
+    notes="Custom calibration model",
+)
+
+# Use the model
+calibrator = Calibrator(model=model)
+confidence = calibrator.calibrate(candidate)
+```
+
+### Integration with Resolver Pipeline
+
+Typical usage within the resolver cascade:
+
+```python
+from resolvekit.calibration import Calibrator, load_calibration_model
+from pathlib import Path
+
+# Initialize calibrator at resolver startup
+model = load_calibration_model(Path("data/base/calibration_model.json"))
+calibrator = Calibrator(model=model)
+
+# During resolution, matchers populate candidate.features
+# Then calibrator converts to confidence probabilities
+
+def resolve(query: str) -> list[tuple[Candidate, float]]:
+    # 1. Matchers generate candidates with features
+    candidates = run_matcher_cascade(query)
+
+    # 2. Calibrator converts to confidence probabilities
+    confidences = calibrator.calibrate_batch(candidates)
+
+    # 3. Return ranked results
+    results = list(zip(candidates, confidences))
+    results.sort(key=lambda x: x[1], reverse=True)  # Sort by confidence
+
+    return results
+```
+
+### Heuristic Scoring Rules
+
+When using heuristic mode (no model), confidence scores follow these rules:
+
+- **Tier 1 - Exact Matches** (0.85-0.95):
+  - Exact code match: 0.95
+  - Canonical name exact: 0.90
+  - Alias exact: 0.85
+
+- **Tier 2 - Fuzzy Matches** (0.5-0.8):
+  - Base: 0.5 + (fuzzy_score × 0.3)
+  - Example: fuzzy_score=0.78 → confidence=0.734
+
+- **Tier 3 - FTS Only** (0.3-0.6):
+  - Base: 0.3 + (fts_score × 0.3)
+  - Example: fts_score=0.7 → confidence=0.51
+
+- **Fallback**: Returns raw matcher score if no features match
+
+### Performance Considerations
+
+```python
+# Batch processing is faster for multiple candidates
+import time
+
+# Single-item processing
+start = time.time()
+for candidate in candidates:
+    confidence = calibrator.calibrate(candidate)
+single_time = time.time() - start
+
+# Batch processing (with numpy)
+start = time.time()
+confidences = calibrator.calibrate_batch(candidates)
+batch_time = time.time() - start
+
+print(f"Single: {single_time:.4f}s")
+print(f"Batch: {batch_time:.4f}s")
+print(f"Speedup: {single_time / batch_time:.1f}x")
+# With numpy: 5-10x speedup for 100+ candidates
+# Without numpy: Similar performance (both use Python loop)
+```
+
+## Implementation Priority
+
+**Phase A** - Core resolver

resolvekit/calibration/__init__.py

@@ -0,0 +1,12 @@
+"""Calibration module for converting scores to probabilities."""
+
+from resolvekit.calibration.calibrator import Calibrator
+from resolvekit.calibration.features import FeatureExtractor
+from resolvekit.calibration.models import CalibrationModel, load_calibration_model
+
+__all__ = [
+    "CalibrationModel",
+    "Calibrator",
+    "FeatureExtractor",
+    "load_calibration_model",
+]

resolvekit/calibration/calibrator.py

@@ -0,0 +1,184 @@
+"""Calibration service for converting scores to probabilities."""
+
+import math
+
+from resolvekit.calibration.features import FeatureExtractor
+from resolvekit.calibration.models import CalibrationModel
+from resolvekit.types import Candidate
+
+try:
+    import numpy as np
+
+    HAS_NUMPY = True
+except ImportError:
+    HAS_NUMPY = False
+
+
+class Calibrator:
+    """
+    Applies calibration models to candidates.
+
+    Supports logistic regression models or heuristic fallback when no model available.
+    """
+
+    def __init__(self, model: CalibrationModel | None = None):
+        """
+        Initialize calibrator.
+
+        Args:
+            model: Optional calibration model. If None, uses heuristic fallback.
+        """
+        self.model = model
+        self.feature_extractor = FeatureExtractor()
+
+    def calibrate(self, candidate: Candidate) -> float:
+        """
+        Calibrate single candidate to confidence probability.
+
+        Args:
+            candidate: Candidate to calibrate
+
+        Returns:
+            Calibrated confidence in [0.0, 1.0]
+        """
+        if self.model:
+            return self._apply_logistic(candidate)
+        else:
+            # Use heuristic fallback
+            return self._apply_heuristic(candidate)
+
+    def _apply_heuristic(self, candidate: Candidate) -> float:
+        """
+        Apply rule-based heuristic scoring.
+
+        Tier 1: Exact matches (0.85-0.95)
+        Tier 2: Fuzzy matches (0.5-0.8)
+        Tier 3: FTS only (0.3-0.6)
+        Fallback: Raw score
+
+        Args:
+            candidate: Candidate to score
+
+        Returns:
+            Heuristic confidence
+        """
+        features = candidate.features
+
+        # Tier 1: Exact matches
+        if features.get("exact_code"):
+            return 0.95
+        if features.get("canonical_exact"):
+            return 0.90
+        if features.get("alias_exact"):
+            return 0.85
+
+        # Tier 2: Fuzzy matches
+        fuzzy_score = features.get("fuzzy_score")
+        if fuzzy_score is not None and fuzzy_score >= 0:
+            return float(0.5 + (fuzzy_score * 0.3))
+
+        # Tier 3: FTS only
+        fts_score = features.get("fts_score")
+        if fts_score is not None and fts_score >= 0:
+            return float(0.3 + (min(fts_score, 1.0) * 0.3))
+
+        # Fallback: raw matcher score
+        return candidate.score
+
+    def _apply_logistic(self, candidate: Candidate) -> float:
+        """
+        Apply logistic regression model: sigmoid(w·x + b).
+
+        Uses Python math.exp (works without numpy).
+
+        Args:
+            candidate: Candidate to calibrate
+
+        Returns:
+            Calibrated probability in [0.0, 1.0]
+        """
+        # Extract features in model's feature order
+        features_dict = self.feature_extractor.extract(candidate)
+        feature_vector = [features_dict.get(f, -1.0) for f in self.model.features]
+
+        # Compute logit: w·x + b
+        logit = (
+            sum(w * x for w, x in zip(self.model.weights, feature_vector, strict=False))
+            + self.model.bias
+        )
+
+        # Clamp extreme values to prevent overflow
+        logit = max(min(logit, 20.0), -20.0)
+
+        # Apply sigmoid: 1 / (1 + exp(-logit))
+        return 1.0 / (1.0 + math.exp(-logit))
+
+    def calibrate_batch(self, candidates: list[Candidate]) -> list[float]:
+        """
+        Calibrate multiple candidates efficiently.
+
+        Uses vectorized numpy operations when available (fast path),
+        falls back to Python loop otherwise (slow path).
+
+        Args:
+            candidates: List of candidates to calibrate
+
+        Returns:
+            List of confidence scores [0.0-1.0] in same order
+        """
+        if not candidates:
+            return []
+
+        if self.model and HAS_NUMPY:
+            # Fast path: vectorized with numpy
+            return self._calibrate_batch_vectorized(candidates)
+        else:
+            # Slow path: Python loop (no numpy or heuristic mode)
+            return [self.calibrate(c) for c in candidates]
+
+    def _calibrate_batch_vectorized(self, candidates: list[Candidate]) -> list[float]:
+        """
+        Vectorized calibration (requires numpy).
+
+        Args:
+            candidates: List of candidates
+
+        Returns:
+            List of confidence scores
+        """
+        # Extract features as numpy array (n_candidates, n_features)
+        feature_matrix = self.feature_extractor.extract_batch(candidates)
+
+        # Build feature matrix in model's feature order
+        # Handle unknown features (not in FEATURE_SCHEMA) by using sentinel -1.0
+        feature_names = list(self.feature_extractor.FEATURE_SCHEMA.keys())
+
+        # Build column indices for known features, None for unknown
+        model_feature_indices = []
+        for f in self.model.features:
+            try:
+                model_feature_indices.append(feature_names.index(f))
+            except ValueError:
+                # Feature not in schema - will use sentinel
+                model_feature_indices.append(None)
+
+        # Build feature matrix for model features (n_candidates, n_model_features)
+        x = np.zeros((feature_matrix.shape[0], len(self.model.features)))
+        for i, idx in enumerate(model_feature_indices):
+            if idx is not None:
+                x[:, i] = feature_matrix[:, idx]
+            else:
+                # Unknown feature - use sentinel -1.0
+                x[:, i] = -1.0
+
+        # Vectorized logistic: sigmoid(x @ w + b)
+        weights = np.array(self.model.weights)
+        logits = x @ weights + self.model.bias
+
+        # Clamp to prevent overflow
+        logits = np.clip(logits, -20.0, 20.0)
+
+        # Sigmoid
+        probabilities = 1.0 / (1.0 + np.exp(-logits))
+
+        return probabilities.tolist()

resolvekit/calibration/features.py

@@ -0,0 +1,139 @@
+"""Feature extraction for calibration."""
+
+from typing import ClassVar
+
+from resolvekit.types import Candidate
+
+# At top of file
+try:
+    import numpy as np
+
+    HAS_NUMPY = True
+except ImportError:
+    HAS_NUMPY = False
+
+
+class FeatureExtractor:
+    """
+    Extracts standardized features from candidates for calibration.
+
+    Reads from candidate.features dict (populated by matchers and constraints)
+    and converts to normalized feature vector with sentinel values for missing data.
+    """
+
+    # Feature schema: feature_name -> expected type
+    FEATURE_SCHEMA: ClassVar[dict[str, type]] = {
+        # Match type features (boolean -> float)
+        "f_exact_code": bool,
+        "f_canonical_exact": bool,
+        "f_alias_exact": bool,
+        # Similarity features (float or None -> float with sentinel)
+        "f_fts_score": float,
+        "f_fts_rank_inv": float,  # Derived from fts_rank
+        "f_edit_similarity": float,
+        "f_trigram_jaccard": float,
+        "f_fuzzy_score": float,
+        # Constraint features (boolean -> float)
+        "f_parent_valid": bool,
+        "f_parent_depth": float,  # Normalized 0-1
+        "f_type_valid": bool,
+        "f_date_valid": bool,
+        "f_membership_valid": bool,
+    }
+
+    def extract(self, candidate: Candidate) -> dict[str, float]:
+        """
+        Extract standardized features from candidate.
+
+        Args:
+            candidate: Candidate with features dict from matchers/constraints
+
+        Returns:
+            Dict of feature_name -> float value
+            Missing numeric features use sentinel -1.0
+            Missing boolean features use 0.0 (False)
+        """
+        raw = candidate.features
+        features = {}
+
+        # Match type features (boolean -> 1.0 or 0.0)
+        features["f_exact_code"] = 1.0 if raw.get("exact_code") else 0.0
+        features["f_canonical_exact"] = 1.0 if raw.get("canonical_exact") else 0.0
+        features["f_alias_exact"] = 1.0 if raw.get("alias_exact") else 0.0
+
+        # Similarity features (with sentinel -1.0 for missing)
+        features["f_fts_score"] = self._get_float(raw, "fts_score", -1.0)
+        features["f_edit_similarity"] = self._get_float(raw, "edit_similarity", -1.0)
+        features["f_trigram_jaccard"] = self._get_float(raw, "trigram_jaccard", -1.0)
+        features["f_fuzzy_score"] = self._get_float(raw, "fuzzy_score", -1.0)
+
+        # FTS rank inverse (1/rank, or -1.0 if missing)
+        fts_rank = raw.get("fts_rank")
+        features["f_fts_rank_inv"] = (
+            1.0 / fts_rank if fts_rank and fts_rank > 0 else -1.0
+        )
+
+        # Constraint features (three-state: -1.0=not checked, 0.0=failed, 1.0=passed)
+        features["f_parent_valid"] = self._get_constraint_feature(raw, "parent_valid")
+        features["f_type_valid"] = self._get_constraint_feature(raw, "type_valid")
+        features["f_date_valid"] = self._get_constraint_feature(raw, "date_valid")
+        features["f_membership_valid"] = self._get_constraint_feature(
+            raw, "membership_valid"
+        )
+
+        # Parent depth normalized (depth / 3, capped at 1.0, or -1.0 if not checked)
+        parent_depth = raw.get("parent_depth")
+        if parent_depth is None:
+            features["f_parent_depth"] = -1.0  # Not checked
+        else:
+            features["f_parent_depth"] = min(parent_depth / 3.0, 1.0)
+
+        return features
+
+    def _get_float(self, raw: dict, key: str, default: float) -> float:
+        """Get float value or default if missing/None."""
+        value = raw.get(key)
+        return float(value) if value is not None else default
+
+    def _get_constraint_feature(self, raw: dict, key: str) -> float:
+        """
+        Get constraint feature with three-state logic.
+
+        Returns:
+            -1.0: Constraint not checked (key missing)
+             0.0: Constraint checked and failed (value is False)
+             1.0: Constraint checked and passed (value is True)
+        """
+        value = raw.get(key)
+        if value is None:
+            return -1.0  # Not checked
+        return 1.0 if value else 0.0  # Checked: True→1.0, False→0.0
+
+    def extract_batch(
+        self, candidates: list[Candidate]
+    ) -> "np.ndarray | list[dict[str, float]]":
+        """
+        Extract features from multiple candidates.
+
+        Args:
+            candidates: List of candidates
+
+        Returns:
+            If numpy available: numpy array of shape (n_candidates, n_features)
+            If numpy unavailable: list of feature dicts
+        """
+        if not candidates:
+            return np.array([]) if HAS_NUMPY else []
+
+        # Extract features for all candidates
+        feature_dicts = [self.extract(c) for c in candidates]
+
+        if not HAS_NUMPY:
+            return feature_dicts
+
+        # Convert to numpy array (all dicts have same keys in same order)
+        feature_names = list(self.FEATURE_SCHEMA.keys())
+        feature_matrix = np.array(
+            [[fd[fname] for fname in feature_names] for fd in feature_dicts]
+        )
+        return feature_matrix