hsds-entity-resolution 2.0.0__tar.gz → 2.2.0__tar.gz

{hsds_entity_resolution-2.0.0 → hsds_entity_resolution-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hsds-entity-resolution
-Version: 2.0.0
+Version: 2.2.0
 Summary: HSDS entity resolution: candidate generation, scoring, a Pair Judge contract and clustering, with a Dagster component.
 Project-URL: Homepage, https://github.com/211-Connect/hsds-entity-resolution
 Project-URL: Repository, https://github.com/211-Connect/hsds-entity-resolution
@@ -48,7 +48,7 @@ runs an incremental seven-stage entity-resolution pipeline on your HSDS data:
 | Stage | What happens |
 |---|---|
 | **Clean entities** | Normalize contact fields, compute content hashes, detect adds/changes/removals since the last run |
-| **Generate candidates** | Block on overlap signals (email, phone, domain, taxonomy, location) to produce candidate pairs |
+| **Generate candidates** | Admit a pair when it shares an Informative Key (name, phone, email, website, address) or its embedding similarity reaches one floor, unless a caller's Structural Exclusion vetoes it |
 | **Score candidates** | Weighted combination of deterministic overlap signals and NLP fuzzy name/description matching |
 | **Judge pairs** *(optional)* | A caller-supplied `PairJudge` answers Same Site, Same Offering, physical delivery and Organization Relation as probabilities for every scored pair |
 | **Apply mitigation** | Carry forward stable pairs, retire pairs for removed entities, detect pair identity continuity |
@@ -211,8 +211,8 @@ Key thresholds:
 | `scoring.deterministic_section_weight` | `0.5625` | `0.50` | Weight of overlap-signal section |
 | `scoring.nlp_section_weight` | `0.4375` | `0.50` | Weight of NLP fuzzy-match section; the two must sum to 1 |
 | `scoring.nlp.fuzzy_threshold` | `0.88` | `0.86` | Minimum name similarity to count as NLP match |
-| `blocking.similarity_threshold` | `0.75` | `0.75` | Minimum embedding cosine similarity for blocking |
-| `blocking.max_candidates_per_entity` | `50` | `125` | Maximum candidate pairs per entity |
+| `blocking.similarity_threshold` | `0.75` | `0.75` | The embedding floor: a pair at or above it is a candidate on similarity alone |
+| `blocking.max_candidates_per_entity` | `50` | `125` | Maximum embedding-floor candidates per anchor (shared-key candidates are not capped by it) |
 
 ---
 
@@ -276,6 +276,44 @@ result.judge_answers  # one row per scored pair, JUDGE_ANSWERS_SCHEMA
 `judge_scored_pairs(...)` runs the same stage on its own, for hosts that run the judge
 as a separate step.
 
+## Candidate admission
+
+One generic rule decides which pairs are scored (2.1.0):
+
+- **Informative Keys.** Two records that share a value of `name`, `phone`, `email`,
+  `website` or `address` (street line required) are a candidate, whatever their
+  embedding similarity, provided the field is informative in both records' source
+  schemas. A field is informative where its values mostly identify a record; a value
+  repeated across hundreds of rows (a category label used as a name) is not.
+- **Embedding floor.** Records at or above `blocking.similarity_threshold` are a
+  candidate, up to `blocking.max_candidates_per_entity` per anchor.
+- **Structural Exclusions.** A caller-supplied callable can veto any pair. Excluded
+  pairs are returned as `excluded_pairs` with the reason, never dropped silently.
+
+Pass both to `run_incremental` (or `run_incremental_until_candidates`):
+
+```python
+from hsds_entity_resolution.core.admission import build_informative_key_table
+
+informative_keys = build_informative_key_table(
+    ratios={"SOURCE_A": {"name": 0.12, "phone": 0.74}},  # distinct-value ratios you measure
+    cutoff=0.3,
+    overrides={"SOURCE_B": {"name": True}},
+)
+
+def exclude(entity_a, entity_b):
+    return "different_by_construction" if ... else None
+
+result = run_incremental(..., informative_keys=informative_keys, structural_exclusion=exclude)
+result.excluded_pairs  # pair_key, entity ids, schemas, similarity, exclusion_reason
+```
+
+Without them every key field is informative and nothing is excluded. Scoring does not
+depend on how a pair was admitted: `pair_rules` still pick scoring overrides from the
+entity type and the two source schemas. `source_policy.admission_rules` and
+`blocking.overlap_prefilter_channels` were removed; passing them fails validation with a
+migration message.
+
 ## Normalisers
 
 `hsds_entity_resolution.normalize` holds pure, idempotent functions on plain strings —

{hsds_entity_resolution-2.0.0 → hsds_entity_resolution-2.2.0}/README.pypi.md

@@ -24,7 +24,7 @@ runs an incremental seven-stage entity-resolution pipeline on your HSDS data:
 | Stage | What happens |
 |---|---|
 | **Clean entities** | Normalize contact fields, compute content hashes, detect adds/changes/removals since the last run |
-| **Generate candidates** | Block on overlap signals (email, phone, domain, taxonomy, location) to produce candidate pairs |
+| **Generate candidates** | Admit a pair when it shares an Informative Key (name, phone, email, website, address) or its embedding similarity reaches one floor, unless a caller's Structural Exclusion vetoes it |
 | **Score candidates** | Weighted combination of deterministic overlap signals and NLP fuzzy name/description matching |
 | **Judge pairs** *(optional)* | A caller-supplied `PairJudge` answers Same Site, Same Offering, physical delivery and Organization Relation as probabilities for every scored pair |
 | **Apply mitigation** | Carry forward stable pairs, retire pairs for removed entities, detect pair identity continuity |
@@ -187,8 +187,8 @@ Key thresholds:
 | `scoring.deterministic_section_weight` | `0.5625` | `0.50` | Weight of overlap-signal section |
 | `scoring.nlp_section_weight` | `0.4375` | `0.50` | Weight of NLP fuzzy-match section; the two must sum to 1 |
 | `scoring.nlp.fuzzy_threshold` | `0.88` | `0.86` | Minimum name similarity to count as NLP match |
-| `blocking.similarity_threshold` | `0.75` | `0.75` | Minimum embedding cosine similarity for blocking |
-| `blocking.max_candidates_per_entity` | `50` | `125` | Maximum candidate pairs per entity |
+| `blocking.similarity_threshold` | `0.75` | `0.75` | The embedding floor: a pair at or above it is a candidate on similarity alone |
+| `blocking.max_candidates_per_entity` | `50` | `125` | Maximum embedding-floor candidates per anchor (shared-key candidates are not capped by it) |
 
 ---
 
@@ -252,6 +252,44 @@ result.judge_answers  # one row per scored pair, JUDGE_ANSWERS_SCHEMA
 `judge_scored_pairs(...)` runs the same stage on its own, for hosts that run the judge
 as a separate step.
 
+## Candidate admission
+
+One generic rule decides which pairs are scored (2.1.0):
+
+- **Informative Keys.** Two records that share a value of `name`, `phone`, `email`,
+  `website` or `address` (street line required) are a candidate, whatever their
+  embedding similarity, provided the field is informative in both records' source
+  schemas. A field is informative where its values mostly identify a record; a value
+  repeated across hundreds of rows (a category label used as a name) is not.
+- **Embedding floor.** Records at or above `blocking.similarity_threshold` are a
+  candidate, up to `blocking.max_candidates_per_entity` per anchor.
+- **Structural Exclusions.** A caller-supplied callable can veto any pair. Excluded
+  pairs are returned as `excluded_pairs` with the reason, never dropped silently.
+
+Pass both to `run_incremental` (or `run_incremental_until_candidates`):
+
+```python
+from hsds_entity_resolution.core.admission import build_informative_key_table
+
+informative_keys = build_informative_key_table(
+    ratios={"SOURCE_A": {"name": 0.12, "phone": 0.74}},  # distinct-value ratios you measure
+    cutoff=0.3,
+    overrides={"SOURCE_B": {"name": True}},
+)
+
+def exclude(entity_a, entity_b):
+    return "different_by_construction" if ... else None
+
+result = run_incremental(..., informative_keys=informative_keys, structural_exclusion=exclude)
+result.excluded_pairs  # pair_key, entity ids, schemas, similarity, exclusion_reason
+```
+
+Without them every key field is informative and nothing is excluded. Scoring does not
+depend on how a pair was admitted: `pair_rules` still pick scoring overrides from the
+entity type and the two source schemas. `source_policy.admission_rules` and
+`blocking.overlap_prefilter_channels` were removed; passing them fails validation with a
+migration message.
+
 ## Normalisers
 
 `hsds_entity_resolution.normalize` holds pure, idempotent functions on plain strings —

{hsds_entity_resolution-2.0.0 → hsds_entity_resolution-2.2.0}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "hsds-entity-resolution"
 requires-python = ">=3.10,<3.15"
-version = "2.0.0"
+version = "2.2.0"
 description = "HSDS entity resolution: candidate generation, scoring, a Pair Judge contract and clustering, with a Dagster component."
 readme = "README.pypi.md"
 keywords = ["dagster", "entity-resolution", "hsds", "deduplication"]

{hsds_entity_resolution-2.0.0 → hsds_entity_resolution-2.2.0}/src/hsds_entity_resolution/config/entity_resolution_run_config.py

@@ -8,15 +8,6 @@ from pydantic import BaseModel, ConfigDict, Field, field_validator, model_valida
 
 from hsds_entity_resolution.types.domain import EntityType
 
-_SUPPORTED_BLOCKING_OVERLAP_CHANNELS = {
-    "email",
-    "phone",
-    "website",
-    "taxonomy",
-    "location",
-    "address_exact",
-}
-
 _ML_SECTION_REMOVED_MESSAGE = (
     "The ML scoring section was removed in hsds-entity-resolution 1.2.0: the score is "
     "deterministic plus NLP only, and the shadow/legacy calibration scores are gone. "
@@ -25,6 +16,33 @@ _ML_SECTION_REMOVED_MESSAGE = (
 )
 
 
+_ADMISSION_REPLACED_MESSAGE = (
+    "Per-lane candidate admission was replaced in hsds-entity-resolution 2.1.0 (ISS-2177) "
+    "by one rule: a pair is a candidate when it shares an Informative Key value or its "
+    "embedding similarity is at or above blocking.similarity_threshold, unless a "
+    "Structural Exclusion vetoes it. Remove {keys} from this config and pass "
+    "informative_keys / structural_exclusion to the pipeline instead."
+)
+
+
+def _reject_replaced_admission_keys(data: Any, *, removed: tuple[str, ...]) -> Any:
+    """Raise a migration error when a config still carries replaced admission keys.
+
+    Args:
+        data: Raw mapping handed to a pydantic ``mode="before"`` validator.
+        removed: Key names that no longer exist in this model.
+
+    Returns:
+        ``data`` unchanged when none of the removed keys are present.
+    """
+    if not isinstance(data, dict):
+        return data
+    present = sorted(key for key in removed if key in data)
+    if present:
+        raise ValueError(_ADMISSION_REPLACED_MESSAGE.format(keys=", ".join(present)))
+    return data
+
+
 def _reject_removed_ml_keys(data: Any, *, removed: tuple[str, ...]) -> Any:
     """Raise a migration error when a config still carries removed ML keys.
 
@@ -64,36 +82,23 @@ class BaseStrictModel(BaseModel):
 
 
 class BlockingConfig(BaseStrictModel):
-    """Candidate blocking and fanout controls."""
+    """Candidate admission and fanout controls.
+
+    ``similarity_threshold`` is the single embedding floor: a pair at or above it is a
+    candidate on similarity alone. ``max_candidates_per_entity`` caps how many such
+    pairs one anchor keeps; pairs admitted by a shared Informative Key are not capped by
+    it (see ``ChunkingConfig`` for the optional key caps).
+    """
 
     similarity_threshold: float = Field(default=0.75, ge=0.0, le=1.0)
     max_candidates_per_entity: int = Field(default=50, ge=1, le=500)
     blocking_batch_size: int = Field(default=5000, ge=1, le=50000)
-    overlap_prefilter_channels: list[str] = Field(
-        default_factory=lambda: ["email", "phone", "website", "taxonomy", "location"]
-    )
 
-    @field_validator("overlap_prefilter_channels")
+    @model_validator(mode="before")
     @classmethod
-    def validate_overlap_prefilter_channels(cls, values: list[str]) -> list[str]:
-        """Validate overlap prefilter channel selections."""
-        normalized: list[str] = []
-        for value in values:
-            if not isinstance(value, str):
-                message = "overlap_prefilter_channels entries must be strings"
-                raise ValueError(message)
-            normalized_value = value.strip().lower()
-            if normalized_value:
-                normalized.append(normalized_value)
-        unique_values = list(dict.fromkeys(normalized))
-        if not unique_values:
-            message = "overlap_prefilter_channels must contain at least one channel"
-            raise ValueError(message)
-        unsupported = sorted(set(unique_values).difference(_SUPPORTED_BLOCKING_OVERLAP_CHANNELS))
-        if unsupported:
-            message = f"Unsupported overlap prefilter channels: {unsupported!r}"
-            raise ValueError(message)
-        return unique_values
+    def reject_replaced_admission_keys(cls, data: Any) -> Any:
+        """Fail with a migration message when the removed overlap prefilter is passed."""
+        return _reject_replaced_admission_keys(data, removed=("overlap_prefilter_channels",))
 
 
 class ChunkingConfig(BaseStrictModel):
@@ -127,8 +132,8 @@ class ChunkingConfig(BaseStrictModel):
         ge=1,
         le=100_000,
         description=(
-            "When set, cap default contact-overlap pairs retained per anchor. "
-            "Unset preserves prior uncapped contact-overlap expansion."
+            "When set, cap the Informative Key pairs considered per anchor. "
+            "Unset leaves Informative Key admission uncapped."
         ),
     )
     max_contact_index_fanout: int | None = Field(
@@ -136,45 +141,12 @@ class ChunkingConfig(BaseStrictModel):
         ge=1,
         le=100_000,
         description=(
-            "When set, truncate inverted-index values that map to more than this many "
-            "entities (deterministic sorted keep). Unset preserves prior uncapped fanout."
+            "When set, truncate Informative Key index values that map to more than this "
+            "many entities (deterministic sorted keep). Unset leaves fanout uncapped."
         ),
     )
 
 
-class AdmissionRuleConfig(BaseStrictModel):
-    """Generic candidate-admission rule evaluated after embedding threshold."""
-
-    rule_id: str
-    entity_types: list[EntityType] = Field(default_factory=lambda: ["organization", "service"])
-    source_relation: Literal[
-        "any",
-        "same_source",
-        "cross_source",
-        "same_profile",
-        "cross_source_same_profile",
-        "cross_profile",
-    ] = "any"
-    source_profiles: list[str] = Field(default_factory=list)
-    min_embedding_similarity: float | None = Field(default=None, ge=0.0, le=1.0)
-    all_of: list[str] = Field(default_factory=list)
-    any_of: list[str] = Field(default_factory=list)
-    none_of: list[str] = Field(default_factory=list)
-
-    @model_validator(mode="after")
-    def validate_admission_rule(self) -> AdmissionRuleConfig:
-        """Validate supported channel names and source-profile references."""
-        referenced = [*self.all_of, *self.any_of, *self.none_of]
-        unsupported = sorted(set(referenced).difference(_SUPPORTED_BLOCKING_OVERLAP_CHANNELS))
-        if unsupported:
-            message = f"Unsupported admission rule channels: {unsupported!r}"
-            raise ValueError(message)
-        if not self.all_of and not self.any_of:
-            message = "Admission rule must define at least one all_of or any_of channel"
-            raise ValueError(message)
-        return self
-
-
 class DeterministicSignalConfig(BaseStrictModel):
     """Configuration for one deterministic overlap signal."""
 
@@ -350,18 +322,27 @@ class SourceProfileConfig(BaseStrictModel):
 
 
 class SourcePolicyConfig(BaseStrictModel):
-    """Generic source-aware policy extension supplied by host applications."""
+    """Source-aware scoring overrides supplied by host applications.
+
+    Only scoring reads it: ``pair_rules`` pick per-pair overrides by entity type and the
+    two records' source profiles. Candidate admission is generic (ISS-2177).
+    """
 
     source_profiles: dict[str, SourceProfileConfig] = Field(default_factory=dict)
-    admission_rules: list[AdmissionRuleConfig] = Field(default_factory=list)
     pair_rules: list[PairRuleConfig] = Field(default_factory=list)
 
+    @model_validator(mode="before")
+    @classmethod
+    def reject_replaced_admission_rules(cls, data: Any) -> Any:
+        """Fail with a migration message when per-lane admission rules are passed."""
+        return _reject_replaced_admission_keys(data, removed=("admission_rules",))
+
     @model_validator(mode="after")
     def validate_source_policy(self) -> SourcePolicyConfig:
-        """Validate rule references against configured source profiles."""
+        """Validate pair-rule references against configured source profiles."""
         profile_ids = set(self.source_profiles)
         referenced: set[str] = set()
-        for rule in [*self.admission_rules, *self.pair_rules]:
+        for rule in self.pair_rules:
             referenced.update(rule.source_profiles)
         unknown = sorted(referenced.difference(profile_ids))
         if unknown:

hsds_entity_resolution-2.2.0/src/hsds_entity_resolution/core/admission.py

@@ -0,0 +1,224 @@
+"""Candidate admission: one generic rule on Informative Keys plus a Structural Exclusion hook.
+
+A pair becomes a candidate when
+
+* its two records share at least one value of an **Informative Key** field, or
+* its embedding similarity is at or above one floor (``blocking.similarity_threshold``),
+
+and the caller's **Structural Exclusion**, if any, does not exclude it.
+
+Whether a field is informative is decided per source schema: a field shared by many
+records of one schema (a service name that is really a category label repeated across
+hundreds of rows) proves nothing when two records share it. The caller measures that,
+usually as a distinct-value ratio, and hands the engine an :data:`InformativeKeyTable`;
+:func:`build_informative_key_table` builds one from ratios, a cutoff and overrides. A key
+admits a pair only when the field is informative in both records' schemas.
+
+Structural Exclusions are pairs that are different by construction (for example two
+records of one site that a source system always splits by category). The engine does
+not know any source system: the caller passes a callable, and every pair it excludes is
+returned with its reason instead of being dropped silently.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Mapping
+from typing import Any, Literal, TypeAlias, get_args
+
+from hsds_entity_resolution.core.dataframe_utils import clean_string_list, clean_text_scalar
+from hsds_entity_resolution.core.score_candidates import (
+    _first_present,  # pyright: ignore[reportPrivateUsage]
+    _normalize_address_component,  # pyright: ignore[reportPrivateUsage]
+)
+
+KeyField: TypeAlias = Literal["name", "phone", "email", "website", "address"]
+KEY_FIELDS: tuple[KeyField, ...] = get_args(KeyField)
+
+InformativeKeyTable: TypeAlias = Mapping[str, Mapping[KeyField, bool]]
+"""Source schema (any case) → key field → whether the field is informative there.
+
+A schema missing from the table, or a field missing for a schema, counts as informative.
+"""
+
+StructuralExclusion: TypeAlias = Callable[[Mapping[str, Any], Mapping[str, Any]], str | None]
+"""``(entity_a, entity_b) -> reason`` when the pair is different by construction, else ``None``."""
+
+KeyValueFilter: TypeAlias = Callable[[Mapping[str, Any], KeyField, set[str]], set[str]]
+"""``(entity, field, values) -> values`` narrowing one record's key values before matching.
+
+For values that never identify a record in a given source (a service name that is the
+record's own category label, an address of a placeholder location). It can only remove
+values: anything it returns that ``key_values`` did not produce is ignored.
+"""
+
+KeyCorroboration: TypeAlias = Callable[[Mapping[str, Any], Mapping[str, Any], KeyField], bool]
+"""``(entity_a, entity_b, field) -> admits`` deciding whether a shared key admits a pair.
+
+For a key that identifies a record only in context (a program name that repeats across
+agencies and identifies a service only near the other one). Consulted once per shared
+field of a key-sharing pair; a field it rejects does not admit the pair. It never affects
+the embedding floor, and a pair can still be admitted by another shared field.
+"""
+
+EMBEDDING_FLOOR_RULE_ID = "embedding_floor"
+EMBEDDING_FLOOR_REASON_CODE = "embedding_floor"
+INFORMATIVE_KEY_RULE_PREFIX = "informative_key:"
+
+
+def informative_key_reason_code(field: KeyField) -> str:
+    """Return the candidate reason code naming the Informative Key that admitted a pair.
+
+    Args:
+        field: The key field whose shared value admitted the pair.
+
+    Returns:
+        For example ``"informative_key_phone"``.
+    """
+    return f"informative_key_{field}"
+
+
+def informative_key_rule_id(fields: set[KeyField]) -> str:
+    """Return the ``blocking_rule_id`` for a pair admitted by shared Informative Keys.
+
+    Args:
+        fields: Every key field the pair shares a value of.
+
+    Returns:
+        For example ``"informative_key:email+phone"``.
+    """
+    return INFORMATIVE_KEY_RULE_PREFIX + "+".join(sorted(fields))
+
+
+def build_informative_key_table(
+    *,
+    ratios: Mapping[str, Mapping[KeyField, float | None]],
+    cutoff: float,
+    overrides: Mapping[str, Mapping[KeyField, bool]],
+) -> dict[str, dict[KeyField, bool]]:
+    """Decide per schema which key fields are informative.
+
+    A field is informative when its distinct-value ratio in the schema is strictly above
+    ``cutoff``; a field with no measured ratio (no values) is not. An override for the
+    schema and field wins over the ratio.
+
+    Args:
+        ratios: Schema → field → distinct-value ratio in ``[0, 1]`` or ``None``.
+        cutoff: The global cutoff.
+        overrides: Schema → field → forced informative yes/no.
+
+    Returns:
+        A table keyed by upper-cased schema with every :data:`KEY_FIELDS` entry set.
+    """
+    upper_overrides = {schema.strip().upper(): fields for schema, fields in overrides.items()}
+    table: dict[str, dict[KeyField, bool]] = {}
+    for schema, schema_ratios in ratios.items():
+        key = schema.strip().upper()
+        schema_overrides = upper_overrides.get(key, {})
+        decided: dict[KeyField, bool] = {}
+        for field in KEY_FIELDS:
+            if field in schema_overrides:
+                decided[field] = schema_overrides[field]
+                continue
+            ratio = schema_ratios.get(field)
+            decided[field] = ratio is not None and ratio > cutoff
+        table[key] = decided
+    return table
+
+
+def is_informative(table: InformativeKeyTable | None, *, schema: str, field: KeyField) -> bool:
+    """Return whether ``field`` is an Informative Key in ``schema``.
+
+    Args:
+        table: The run's table, or ``None`` when the caller supplied none.
+        schema: Source schema of the record.
+        field: Key field.
+
+    Returns:
+        ``True`` unless the table says the field is not informative in that schema.
+    """
+    if table is None:
+        return True
+    schema_table = _schema_entry(table, schema)
+    if schema_table is None or field not in schema_table:
+        return True
+    return bool(schema_table[field])
+
+
+def _schema_entry(table: InformativeKeyTable, schema: str) -> Mapping[KeyField, bool] | None:
+    """Look up a schema case-insensitively."""
+    if schema in table:
+        return table[schema]
+    upper = schema.strip().upper()
+    for candidate, entry in table.items():
+        if candidate.strip().upper() == upper:
+            return entry
+    return None
+
+
+def key_values(entity: Mapping[str, Any], field: KeyField) -> set[str]:
+    """Return the comparable values of one key field for one record.
+
+    Values are compared exactly after light cleaning (lowercase, trimmed, whitespace
+    collapsed); an address needs a street line and becomes ``street|city|state|zip``.
+
+    Args:
+        entity: A cleaned entity row.
+        field: Key field.
+
+    Returns:
+        The record's values for the field; empty when it has none.
+    """
+    if field == "name":
+        name = clean_text_scalar(entity.get("name"))
+        return {name} if name else set()
+    if field == "phone":
+        return set(clean_string_list(entity.get("phones")))
+    if field == "email":
+        return set(clean_string_list(entity.get("emails")))
+    if field == "website":
+        return set(clean_string_list(entity.get("websites")))
+    return set(_address_values(entity.get("locations")))
+
+
+def filtered_key_values(
+    entity: Mapping[str, Any], field: KeyField, key_value_filter: KeyValueFilter | None
+) -> set[str]:
+    """Return a record's key values after the caller's optional filter.
+
+    Args:
+        entity: A cleaned entity row.
+        field: Key field.
+        key_value_filter: The caller's filter, or ``None``.
+
+    Returns:
+        The values that may admit a pair; never more than :func:`key_values` returns.
+    """
+    values = key_values(entity, field)
+    if key_value_filter is None or not values:
+        return values
+    return values & key_value_filter(entity, field, set(values))
+
+
+def _address_values(locations_value: Any) -> list[str]:
+    """Build exact-address tokens, requiring a street line."""
+    if not isinstance(locations_value, list):
+        return []
+    output: list[str] = []
+    for location in locations_value:
+        if not isinstance(location, dict):
+            continue
+        street = _normalize_address_component(
+            _first_present(location, ("address_1", "address1", "line1", "street", "address"))
+        )
+        if not street:
+            continue
+        parts = [
+            street,
+            _normalize_address_component(_first_present(location, ("city",))),
+            _normalize_address_component(_first_present(location, ("state",))),
+            _normalize_address_component(
+                _first_present(location, ("postal_code", "postal", "zip", "zipcode"))
+            ),
+        ]
+        output.append("|".join(part for part in parts if part))
+    return clean_string_list(output)

{hsds_entity_resolution-2.0.0 → hsds_entity_resolution-2.2.0}/src/hsds_entity_resolution/core/domain_utils.py

@@ -61,23 +61,6 @@ def extract_contact_domains(*, emails_value: object, websites_value: object) ->
     return domains
 
 
-def extract_gov_website_registered_domains(*, websites_value: object) -> set[str]:
-    """Extract registrable ``.gov`` domains from website values only.
-
-    This intentionally ignores emails. Candidate generation uses it as a
-    conservative URL-only bypass for public-sector services, while ordinary
-    email/website domain scoring continues to use ``extract_contact_domains``.
-    """
-    domains: set[str] = set()
-    for website in clean_string_list(websites_value):
-        if "@" in website or website.startswith("mailto:"):
-            continue
-        token = _parse_domain_token(website, is_email=False)
-        if token is not None and token.registered_domain.endswith(".gov"):
-            domains.add(token.registered_domain)
-    return domains
-
-
 def domain_overlap_score(
     *,
     left_emails: object,