hsds-entity-resolution 2.0.0__tar.gz

hsds_entity_resolution-2.0.0/.gitignore

@@ -0,0 +1,234 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+#poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+#pdm.lock
+#pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+#pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+.uv-cache/
+.cursor/
+
+# PyPI configuration file
+.pypirc
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Streamlit
+.streamlit/secrets.toml
+
+
+#Ignore cursor AI rules
+.cursor/rules/codacy.mdc
+
+# Codacy
+.codacy/AGENTS.md
+
+# dbt — target/ contains per-run artifacts; never commit.
+consumer/dbt/target/
+hsds_dbt_macros/target/
+
+.dg/
+
+consumer/
+hsds_dbt_macros/
+scripts/
+
+HSDS_DATA_MODEL.md
+_hsds_table_desc/
+
+.agents/skills/snowsql-investigation/
+.codacy/
+docs/
+!docs/
+docs/*
+!docs/deduplication-schema-audit.md
+.DS_Store

hsds_entity_resolution-2.0.0/PKG-INFO

@@ -0,0 +1,306 @@
+Metadata-Version: 2.5
+Name: hsds-entity-resolution
+Version: 2.0.0
+Summary: HSDS entity resolution: candidate generation, scoring, a Pair Judge contract and clustering, with a Dagster component.
+Project-URL: Homepage, https://github.com/211-Connect/hsds-entity-resolution
+Project-URL: Repository, https://github.com/211-Connect/hsds-entity-resolution
+Project-URL: Issues, https://github.com/211-Connect/hsds-entity-resolution/issues
+Keywords: dagster,deduplication,entity-resolution,hsds
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Typing :: Typed
+Requires-Python: <3.15,>=3.10
+Requires-Dist: dagster<1.13,>=1.11
+Requires-Dist: numpy<3,>=1.26
+Requires-Dist: polars<2,>=1.18
+Requires-Dist: pydantic<3,>=2
+Requires-Dist: tldextract>=5.3.1
+Description-Content-Type: text/markdown
+
+# hsds-entity-resolution
+
+Reusable [Dagster](https://dagster.io) component for incremental entity resolution on
+[HSDS](https://docs.openreferral.org/en/latest/) (Human Services Data Specification) records.
+
+**Install:**
+
+```bash
+pip install hsds-entity-resolution
+```
+
+---
+
+## What this package does
+
+Community 211 directories and social-services platforms often ingest the same organizations
+and services from multiple data partners. Over time those records diverge — different spellings,
+phone formats, partial addresses — making it hard to know which rows describe the same real-world
+entity.
+
+`hsds-entity-resolution` provides a single Dagster component, `EntityResolutionComponent`, that
+runs an incremental seven-stage entity-resolution pipeline on your HSDS data:
+
+| Stage | What happens |
+|---|---|
+| **Clean entities** | Normalize contact fields, compute content hashes, detect adds/changes/removals since the last run |
+| **Generate candidates** | Block on overlap signals (email, phone, domain, taxonomy, location) to produce candidate pairs |
+| **Score candidates** | Weighted combination of deterministic overlap signals and NLP fuzzy name/description matching |
+| **Judge pairs** *(optional)* | A caller-supplied `PairJudge` answers Same Site, Same Offering, physical delivery and Organization Relation as probabilities for every scored pair |
+| **Apply mitigation** | Carry forward stable pairs, retire pairs for removed entities, detect pair identity continuity |
+| **Cluster pairs** | Greedy correlation clustering groups high-confidence duplicate pairs into clusters |
+| **Materialize review queue** | Pairs that score above the *maybe* threshold but below *duplicate* are surfaced for human review |
+| **Prepare artifacts** | Package all stage outputs into a typed bundle ready for downstream persistence |
+
+The component is **storage-agnostic**: it accepts and emits Polars DataFrames, so you wire it
+into whatever persistence layer your Dagster project uses.
+
+---
+
+## Prerequisites
+
+- Python 3.10–3.13
+- A [Dagster](https://dagster.io) project managed with the
+  [`dg` CLI](https://docs.dagster.io/guides/build/projects-and-components/components/)
+- Four upstream Dagster assets that supply denormalized HSDS entity data (see [Input assets](#input-assets))
+
+---
+
+## Quickstart
+
+### 1. Discover the component
+
+```bash
+dg list components --package hsds_entity_resolution
+```
+
+### 2. Add to your component YAML
+
+```yaml
+type: hsds_entity_resolution.dagster.components.EntityResolutionComponent
+attributes:
+  team_id: my_team
+  scope_id: regional
+  entity_type: organization
+```
+
+That's enough to wire the component into your Dagster asset graph. The four required upstream
+assets default to the keys `organization_entities`, `service_entities`,
+`previous_entity_index`, and `previous_pair_state_index` — override them with the
+`*_asset_key` attributes if your project uses different names.
+
+---
+
+## Input assets
+
+The component depends on four upstream assets. Each must yield a Polars `DataFrame`
+(or be coercible to one).
+
+### `organization_entities` and `service_entities`
+
+Denormalized HSDS entity records. Required columns:
+
+| Column | Type | Description |
+|---|---|---|
+| `entity_id` | `str` | Stable unique identifier for this record |
+| `source_schema` | `str` | Tenant or source identifier (e.g. `region_a`) |
+| `name` | `str` | Entity name used for NLP scoring |
+| `description` | `str` | Entity description used for NLP scoring |
+| `emails` | `list[str]` | Normalized email addresses |
+| `phones` | `list[str]` | Normalized phone numbers |
+| `websites` | `list[str]` | Normalized domain names |
+| `locations` | `list[dict]` | Structured address/location objects |
+| `taxonomies` | `list[dict]` | AIRS taxonomy code objects |
+| `identifiers` | `list[dict]` | External identifier objects |
+| `embedding_vector` | `list[float]` | Pre-computed text embedding (e.g. 384-dim) |
+
+Optional passthrough columns (`display_name`, `display_description`, `alternate_name`,
+`short_description`, `application_process`, `fees_description`, `eligibility_description`,
+`resource_writer_name`, `assured_date`, `assurer_email`, `original_id`,
+`organization_original_id`, `organization_name`, `organization_id`,
+`services_rollup`) are forwarded unchanged to the normalized entity cache and
+persistence artifacts.
+
+### `previous_entity_index`
+
+The `normalized_organization` or `normalized_service` output from the *previous* run of this
+component. Pass an empty `pl.DataFrame()` on first run.
+
+### `previous_pair_state_index`
+
+The `pair_state_index` output from the *previous* run of this component. Pass an empty
+`pl.DataFrame()` on first run. The component uses this to carry forward stable pair scores
+and detect pair identity continuity across incremental runs.
+
+---
+
+## Component attributes
+
+| Attribute | Type | Default | Description |
+|---|---|---|---|
+| `team_id` | `str` | `"hsds"` | Team identifier written into run metadata |
+| `scope_id` | `str` | `"default"` | Deployment or region identifier |
+| `entity_type` | `"organization" \| "service"` | `"organization"` | Which entity type this instance processes |
+| `policy_version` | `str` | `"hsds-er-v1"` | Scoring policy version tag |
+| `model_version` | `str` | `"embedding-only-v1"` | Embedding model version tag |
+| `explicit_backfill` | `bool` | `False` | Force a full re-run even when no entity changes are detected |
+| `organization_entities_asset_key` | `str` | `"organization_entities"` | Asset key for upstream org entities |
+| `service_entities_asset_key` | `str` | `"service_entities"` | Asset key for upstream service entities |
+| `previous_entity_index_asset_key` | `str` | `"previous_entity_index"` | Asset key for previous-run entity cache |
+| `previous_pair_state_index_asset_key` | `str` | `"previous_pair_state_index"` | Asset key for previous-run pair state |
+| `output_asset_prefix` | `list[str]` | `[]` | Prefix prepended to all output asset keys |
+| `constants_overrides` | `dict` | `{}` | Deep-merge override for scoring constants (see [Tuning](#tuning-scoring-constants)) |
+
+---
+
+## Output assets
+
+The component emits fifteen assets. Using `output_asset_prefix: ["my_prefix"]` namespaces
+all keys under `my_prefix/`.
+
+| Asset key | Description |
+|---|---|
+| `normalized_organization` | Cleaned, normalized organization records with content hashes |
+| `normalized_service` | Cleaned, normalized service records with content hashes |
+| `entity_delta_summary` | Single-row summary of added / changed / removed entity counts |
+| `removed_entity_ids` | Entity IDs that were present last run but are absent this run |
+| `candidate_pairs` | Raw blocked candidate pairs before scoring |
+| `scored_pairs` | Candidate pairs with composite scores and tier labels (`duplicate`, `maybe`, `non_duplicate`) |
+| `pair_reasons` | Per-feature evidence rows for every scored pair |
+| `mitigation_events` | Pairs reassigned due to entity changes between runs |
+| `removed_pair_ids` | Pair IDs retired this run (entity removed or score dropped) |
+| `pair_id_remap` | Maps old pair IDs to new ones when pair identity shifts |
+| `clusters` | Entity groups identified as duplicates via correlation clustering |
+| `cluster_pairs` | Edges within each cluster |
+| `pair_state_index` | Full pair state — feed this back as `previous_pair_state_index` next run |
+| `review_queue_items` | Borderline pairs above the *maybe* threshold, ready for human review |
+| `run_summary` | Single-row run metrics (candidate count, duplicate count, cluster count, etc.) |
+| `persistence_artifact_bundle` | Dict containing all stage outputs packaged for bulk persistence |
+
+---
+
+## Tuning scoring constants
+
+The default thresholds are calibrated for HSDS data. Override any constant via
+`constants_overrides` without subclassing:
+
+```yaml
+type: hsds_entity_resolution.dagster.components.EntityResolutionComponent
+attributes:
+  team_id: my_team
+  scope_id: regional
+  entity_type: organization
+  constants_overrides:
+    scoring:
+      duplicate_threshold: 0.85
+      maybe_threshold: 0.70
+      nlp:
+        fuzzy_threshold: 0.90
+```
+
+Key thresholds:
+
+| Constant | Default (org) | Default (service) | Description |
+|---|---|---|---|
+| `scoring.duplicate_threshold` | `0.82` | `0.70` | Minimum score to auto-cluster as duplicate |
+| `scoring.maybe_threshold` | `0.68` | `0.62` | Minimum score to send to review queue |
+| `scoring.deterministic_section_weight` | `0.5625` | `0.50` | Weight of overlap-signal section |
+| `scoring.nlp_section_weight` | `0.4375` | `0.50` | Weight of NLP fuzzy-match section; the two must sum to 1 |
+| `scoring.nlp.fuzzy_threshold` | `0.88` | `0.86` | Minimum name similarity to count as NLP match |
+| `blocking.similarity_threshold` | `0.75` | `0.75` | Minimum embedding cosine similarity for blocking |
+| `blocking.max_candidates_per_entity` | `50` | `125` | Maximum candidate pairs per entity |
+
+---
+
+## Multiple instances
+
+Deploy one component instance per entity type and scope:
+
+```yaml
+# organizations
+type: hsds_entity_resolution.dagster.components.EntityResolutionComponent
+attributes:
+  team_id: my_team
+  scope_id: regional
+  entity_type: organization
+  output_asset_prefix: ["org_er"]
+
+---
+
+# services
+type: hsds_entity_resolution.dagster.components.EntityResolutionComponent
+attributes:
+  team_id: my_team
+  scope_id: regional
+  entity_type: service
+  output_asset_prefix: ["svc_er"]
+```
+
+---
+
+## Pair Judge contract
+
+The engine fixes what a judge reads and what it answers; it ships no questions or model.
+
+- `hsds_entity_resolution.judge.PairState`: both records as HSDS text (name, alternate
+  name, description, short description, eligibility, fees, application process,
+  taxonomy names, sites, phones as digits, websites, organization name and
+  description), a code-computed site comparison word (`same address`, `same city`,
+  `different`, `unknown`), the prior score as a bucket in words, and one free-text
+  `source_profile` slot per side. The engine fills the HSDS fields and leaves the
+  profile slots empty unless you pass `source_profiles={source_schema: text}`.
+- `hsds_entity_resolution.judge.PairAnswers`: `same_site`, `same_offering`,
+  `physically_delivered` (probabilities; the last two are null for organization
+  pairs), and `organization_relation` as a distribution over `same`,
+  `parent_and_chapter`, `affiliated`, `unrelated`, `cannot_tell`, plus `model_id` and
+  `question_set_version`. The judge never answers "duplicate"; compose that in your
+  own code.
+- `hsds_entity_resolution.judge.PairJudge`: a protocol with `judge(states) -> answers`
+  and a `state_token_budget`. Size states for the smallest model you may switch to
+  (`SMALLEST_MODEL_STATE_BUDGET_TOKENS`, 32k).
+- `ReferenceJudge` answers 0.5 everywhere and a uniform relation, so the pipeline runs
+  end to end without a model.
+
+```python
+from hsds_entity_resolution.core import run_incremental
+from hsds_entity_resolution.judge import ReferenceJudge
+
+result = run_incremental(..., judge=ReferenceJudge(), source_profiles={"source_a": "..."})
+result.judge_answers  # one row per scored pair, JUDGE_ANSWERS_SCHEMA
+```
+
+`judge_scored_pairs(...)` runs the same stage on its own, for hosts that run the judge
+as a separate step.
+
+## Normalisers
+
+`hsds_entity_resolution.normalize` holds pure, idempotent functions on plain strings —
+`normalize_phone` (digits plus extension), `normalize_email`, `normalize_url`,
+`normalize_address_component`, `normalize_postal_code` — usable outside the engine.
+
+## Migrating from `hsds-record-matcher` 1.x
+
+- **Install name.** The distribution is now `hsds-entity-resolution`, matching the
+  import name `hsds_entity_resolution`. Uninstall `hsds-record-matcher` first.
+- **ML scoring section removed (1.2.0).** `scoring.ml`, `scoring.calibration`,
+  `scoring.ml_section_weight` and pair-rule `ml_section_weight` / `ml_gate_threshold`
+  now fail validation. `deterministic_section_weight + nlp_section_weight` must equal
+  1.0; to keep your scores, divide each by their old sum. Scored pairs no longer carry
+  `ml_section_score`, `legacy_confidence_score`, `shadow_confidence_score`,
+  `shadow_log_odds` or `calibration_version`, and `run_incremental` no longer takes
+  `taxonomy_embeddings`.
+- **Judge stage (2.0.0).** `run_incremental` accepts `judge` and `source_profiles`;
+  without a judge every output is unchanged.
+- **Logging.** Core modules log through `logging.getLogger("hsds_entity_resolution...")`
+  instead of Dagster's logger. Add `hsds_entity_resolution` to your Dagster
+  `python_logs.managed_python_loggers` to see them in the Dagster UI.
+
+## Links
+
+- **Source:** [github.com/211-Connect/hsds-entity-resolution](https://github.com/211-Connect/hsds-entity-resolution)
+- **Issues:** [github.com/211-Connect/hsds-entity-resolution/issues](https://github.com/211-Connect/hsds-entity-resolution/issues)
+- **PyPI:** [pypi.org/project/hsds-entity-resolution](https://pypi.org/project/hsds-entity-resolution/)