mismo 0.3.0__tar.gz

mismo-0.3.0/PKG-INFO

@@ -0,0 +1,111 @@
+Metadata-Version: 2.3
+Name: mismo
+Version: 0.3.0
+Summary: The SQL/Ibis powered sklearn of record linkage.
+Keywords: record linkage,entity resolution,fuzzy linking,machine learning,ibis,sql,splink,duckdb
+Author: Nick Crews
+Author-email: Nick Crews <nicholas.b.crews@gmail.com>
+License: LGPL-3.0-or-later
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Requires-Dist: ibis-framework>=9.1.0
+Requires-Dist: sqlglot>=25.29.0
+Requires-Dist: typing-extensions>=4.0.0 ; python_full_version < '3.11'
+Requires-Dist: scikit-learn>=1.5.2 ; extra == 'metrics'
+Requires-Dist: postal>=1.1.7,<1.1.11 ; (sys_platform == 'darwin' and extra == 'postal') or (sys_platform == 'linux' and extra == 'postal')
+Requires-Dist: en-us-address-ner-sm ; extra == 'spacy'
+Requires-Dist: spacy>=3.8.2 ; extra == 'spacy'
+Requires-Dist: altair>=5.0.0 ; extra == 'viz'
+Requires-Dist: ipywidgets>=7.5.1 ; extra == 'viz'
+Requires-Dist: solara-ui>=1.51.0 ; extra == 'viz'
+Requires-Dist: anywidget>=0.9.18 ; extra == 'viz'
+Requires-Python: >=3.10
+Project-URL: Documentation, https://nickcrews.github.io/mismo
+Project-URL: Homepage, https://github.com/NickCrews/mismo
+Project-URL: Issues, https://github.com/NickCrews/mismo/issues
+Project-URL: Source, https://github.com/NickCrews/mismo
+Provides-Extra: metrics
+Provides-Extra: postal
+Provides-Extra: spacy
+Provides-Extra: viz
+Description-Content-Type: text/markdown
+
+# Mismo
+
+[![PyPI - Version](https://img.shields.io/pypi/v/mismo.svg)](https://pypi.org/project/mismo)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/mismo.svg)](https://pypi.org/project/mismo)
+
+The SQL/Ibis powered sklearn of record linkage.
+
+Still in alpha stage. Breaking changes will happen frequently
+and with no warning. Once things are more stabilized I
+will come up with a stability policy. Any suggestions as
+to how you want the API to look like would be greatly appreciated.
+I do use this in my work, so at least I do decent job of
+ensuring correctness.
+
+-----
+
+## Goals
+
+Mismo tries to be the sklearn of record linkage, backed by the scalability
+and power of SQL and [Ibis](https://ibis-project.org/). It is made of many small
+data structures and functions, each with a well-defined and standard API
+that allows them to be composed together and extended easily.
+None of the other record linkage packages I have seen, such as
+[Splink](https://github.com/moj-analytical-services/splink),
+[Dedupe](https://www.github.com/dedupeio/dedupe), or
+[Record Linkage Toolkit](https://github.com/J535D165/recordlinkage),
+had all of these properties, so I decided to make my own.
+
+See [Goals and Alternatives](https://nickcrews.github.io/mismo/concepts/goals_and_alternatives)
+for a more detailed discussion of the goals of Mismo and how it compares to other
+record linkage packages.
+
+## Features
+- Supports larger-than-memory datasets, executed on powerful SQL engines.
+  Use DuckDB for prototyping and for jobs up to maybe ~10M records,
+  or Spark or other distributed backends for larger tasks, without
+  needing to change your code!
+- Use the clean, strong-typed, pythonic, Dataframe APIs of [Ibis](https://ibis-project.org/).
+- Small, modular functions and data structures that are easy to plug together
+  and extend.
+- Layered API: Use top-level APIs if your task is common enough that it is
+  supported out of the box.
+
+## Installation
+
+[`mismo` is available on PyPI](https://pypi.org/project/mismo/).
+I try to publish semver'ed releases after most changes.
+
+If I forget to do this, then there are also[prereleases on PyPI](https://pypi.org/project/mismo/#history).
+These are published every week by a github action using the HEAD commit of this repo.
+
+You can also install directly from a branch or a specific commit from github:
+
+```console
+uv pip install "mismo[viz] @ git+https://github.com/NickCrews/mismo@<SOME-SHA-OR-BRANCH>"
+```
+
+## Examples
+
+See the [example notebook](https://nickcrews.github.io/mismo/examples/patent_deduplication).
+
+## Documentation
+
+See the [documentation](https://nickcrews.github.io/mismo).
+
+## Contributing
+
+See the [contributing guide](https://nickcrews.github.io/mismo/contributing/).
+
+## License
+
+`mismo` is distributed under the terms of the
+[LGPL-3.0-or-later](https://spdx.org/licenses/LGPL-3.0-or-later.html) license.

mismo-0.3.0/README.md

@@ -0,0 +1,73 @@
+# Mismo
+
+[![PyPI - Version](https://img.shields.io/pypi/v/mismo.svg)](https://pypi.org/project/mismo)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/mismo.svg)](https://pypi.org/project/mismo)
+
+The SQL/Ibis powered sklearn of record linkage.
+
+Still in alpha stage. Breaking changes will happen frequently
+and with no warning. Once things are more stabilized I
+will come up with a stability policy. Any suggestions as
+to how you want the API to look like would be greatly appreciated.
+I do use this in my work, so at least I do decent job of
+ensuring correctness.
+
+-----
+
+## Goals
+
+Mismo tries to be the sklearn of record linkage, backed by the scalability
+and power of SQL and [Ibis](https://ibis-project.org/). It is made of many small
+data structures and functions, each with a well-defined and standard API
+that allows them to be composed together and extended easily.
+None of the other record linkage packages I have seen, such as
+[Splink](https://github.com/moj-analytical-services/splink),
+[Dedupe](https://www.github.com/dedupeio/dedupe), or
+[Record Linkage Toolkit](https://github.com/J535D165/recordlinkage),
+had all of these properties, so I decided to make my own.
+
+See [Goals and Alternatives](https://nickcrews.github.io/mismo/concepts/goals_and_alternatives)
+for a more detailed discussion of the goals of Mismo and how it compares to other
+record linkage packages.
+
+## Features
+- Supports larger-than-memory datasets, executed on powerful SQL engines.
+  Use DuckDB for prototyping and for jobs up to maybe ~10M records,
+  or Spark or other distributed backends for larger tasks, without
+  needing to change your code!
+- Use the clean, strong-typed, pythonic, Dataframe APIs of [Ibis](https://ibis-project.org/).
+- Small, modular functions and data structures that are easy to plug together
+  and extend.
+- Layered API: Use top-level APIs if your task is common enough that it is
+  supported out of the box.
+
+## Installation
+
+[`mismo` is available on PyPI](https://pypi.org/project/mismo/).
+I try to publish semver'ed releases after most changes.
+
+If I forget to do this, then there are also[prereleases on PyPI](https://pypi.org/project/mismo/#history).
+These are published every week by a github action using the HEAD commit of this repo.
+
+You can also install directly from a branch or a specific commit from github:
+
+```console
+uv pip install "mismo[viz] @ git+https://github.com/NickCrews/mismo@<SOME-SHA-OR-BRANCH>"
+```
+
+## Examples
+
+See the [example notebook](https://nickcrews.github.io/mismo/examples/patent_deduplication).
+
+## Documentation
+
+See the [documentation](https://nickcrews.github.io/mismo).
+
+## Contributing
+
+See the [contributing guide](https://nickcrews.github.io/mismo/contributing/).
+
+## License
+
+`mismo` is distributed under the terms of the
+[LGPL-3.0-or-later](https://spdx.org/licenses/LGPL-3.0-or-later.html) license.

mismo-0.3.0/mismo/__init__.py

@@ -0,0 +1,55 @@
+from __future__ import annotations
+
+import importlib.metadata
+import warnings
+
+from mismo import arrays as arrays
+from mismo import cluster as cluster
+from mismo import compare as compare
+from mismo import eda as eda
+from mismo import exceptions as exceptions
+from mismo import fs as fs
+from mismo import joins as joins
+from mismo import lib as lib
+from mismo import linkage as linkage
+from mismo import linker as linker
+from mismo import playdata as playdata
+from mismo import sets as sets
+from mismo import text as text
+from mismo import tf as tf
+from mismo import types as types
+from mismo import vector as vector
+from mismo._counts_table import CountsTable as CountsTable
+from mismo._datasets import Datasets as Datasets
+from mismo._explain import explain as explain
+from mismo._n_naive import n_naive_comparisons as n_naive_comparisons
+from mismo._recipe import PRecipe as PRecipe
+from mismo.joins import HasJoinCondition as HasJoinCondition
+from mismo.joins import IntoHasJoinCondition as IntoHasJoinCondition
+from mismo.joins import join as join
+from mismo.joins import join_condition as join_condition
+from mismo.joins import left as left
+from mismo.joins import right as right
+from mismo.linkage import Linkage as Linkage
+from mismo.linker import EmptyLinker as EmptyLinker
+from mismo.linker import FullLinker as FullLinker
+from mismo.linker import IDLinker as IDLinker
+from mismo.linker import JoinLinker as JoinLinker
+from mismo.linker import KeyLinker as KeyLinker
+from mismo.linker import Linker as Linker
+from mismo.linker import OrLinker as OrLinker
+from mismo.linker import empty_linkage as empty_linkage
+from mismo.linker import full_linkage as full_linkage
+from mismo.types import Diff as Diff
+from mismo.types import DiffStats as DiffStats
+from mismo.types import LinkCountsTable as LinkCountsTable
+from mismo.types import LinkedTable as LinkedTable
+from mismo.types import LinksTable as LinksTable
+from mismo.types import UnionTable as UnionTable
+from mismo.types import Updates as Updates
+
+try:
+    __version__ = importlib.metadata.version(__name__)
+except importlib.metadata.PackageNotFoundError as e:
+    warnings.warn(f"Could not determine version of {__name__}\n{e!s}", stacklevel=2)
+    __version__ = "unknown"

mismo-0.3.0/mismo/_common.py

@@ -0,0 +1,28 @@
+from __future__ import annotations
+
+import ibis
+
+
+def check_tables_and_links(
+    left: ibis.Table, right: ibis.Table, links: ibis.Table
+) -> None:
+    if "record_id" not in left.columns:
+        raise ValueError("column 'record_id' not in table")
+    if "record_id" not in right.columns:
+        raise ValueError("column 'record_id' not in other")
+    if "record_id_l" not in links.columns:
+        raise ValueError("column 'record_id_l' not in links")
+    if "record_id_r" not in links.columns:
+        raise ValueError("column 'record_id_r' not in links")
+    try:
+        left.record_id == links.record_id_l
+    except Exception:
+        raise ValueError(
+            f"left.record_id of type {left.record_id.type()} is not comparable with links.record_id_l of type {links.record_id_l.type()}"  # noqa: E501
+        )
+    try:
+        right.record_id == links.record_id_r
+    except Exception:
+        raise ValueError(
+            f"right.record_id of type {right.record_id.type()} is not comparable with links.record_id_r of type {links.record_id_r.type()}"  # noqa: E501
+        )

mismo-0.3.0/mismo/_counts_table.py

@@ -0,0 +1,137 @@
+from __future__ import annotations
+
+import functools
+from typing import TYPE_CHECKING, NamedTuple
+
+import ibis
+from ibis import _
+from ibis.expr import types as ir
+
+from mismo.types._wrapper import TableWrapper
+
+if TYPE_CHECKING:
+    import altair as alt
+
+
+class _HistSpec(NamedTuple):
+    n_title: str
+    chart_title: str
+    chart_subtitle: str
+
+
+class CountsTable(TableWrapper):
+    """A table with at least an Integer column named `n`.
+
+    There will also be variable number of other columns that act as identifiers.
+
+    You won't create this directly, it will be returned to you
+    from eg [KeyLinker.key_counts_left][mismo.KeyLinker.key_counts_left],
+    [KeyLinker.key_counts_right][mismo.KeyLinker.key_counts_right],
+    or [KeyLinker.pair_counts][mismo.KeyLinker.pair_counts].
+    """
+
+    n: ir.IntegerColumn
+    """The column containing the count."""
+
+    # This MUST be set in subclasses
+    _HIST_SPEC: _HistSpec
+
+    @functools.cache
+    def n_total(self) -> int:
+        """n.sum().fill_null(0), just here for convenience."""
+        raw = self.n.sum().execute()
+        return int(raw) if raw is not None else 0
+
+    def chart(self) -> alt.Chart:
+        return _counts_chart(self, hist_spec=self._HIST_SPEC)
+
+
+class KeyCountsTable(CountsTable):
+    _HIST_SPEC = _HistSpec(
+        "Number of Records", "Number of Records by Key", "{n_total:_} Total Records"
+    )
+
+
+class PairCountsTable(CountsTable):
+    _HIST_SPEC = _HistSpec(
+        "Number of Pairs", "Number of Pairs by Key", "{n_total:_} Total Pairs"
+    )
+
+
+def _counts_chart(counts: CountsTable, *, hist_spec: _HistSpec):
+    import altair as alt
+
+    n_total = counts.n_total()
+    key_cols = [c for c in counts.columns if c != "n"]
+    val = "(" + ibis.literal(", ").join(counts[c].cast(str) for c in key_cols) + ")"
+    key_and_n = counts.mutate(val.name("key"), "n")
+    key_and_n = key_and_n.filter(_.n > 0)
+    frac = key_and_n.n / n_total if n_total > 0 else 0
+    key_and_n = key_and_n.mutate(
+        frac=frac,
+        explanation=(
+            "Out of the "
+            + f"{n_total:_}, "
+            + key_and_n.n.cast(str)
+            + " ("
+            + (frac * 100).cast(int).cast(str)
+            + "%) had the key of "
+            + key_and_n.key.cast(str)
+        ),
+    )
+    n_keys_total = key_and_n.count().execute()
+    key_title = "(" + ", ".join(key_cols) + ")"
+    scrubber_selection = alt.selection_interval(encodings=["x"], empty=True)
+    width = 800
+    zoomin = (
+        alt.Chart(key_and_n, width=width)
+        .mark_bar()
+        .encode(
+            alt.X("key:O", title=key_title, sort="-y"),
+            alt.Y(
+                "n:Q",
+                title=hist_spec.n_title,
+                scale=alt.Scale(type="symlog"),
+            ),
+            tooltip=[
+                alt.Tooltip("n:Q", title=hist_spec.n_title, format=","),
+                alt.Tooltip("frac:Q", title="Fraction", format=".2%"),
+                *[alt.Tooltip(col) for col in key_cols],
+                alt.Tooltip("explanation", title="Explanation"),
+            ],
+        )
+        .transform_filter(scrubber_selection)
+    )
+    scrubber = (
+        alt.Chart(
+            key_and_n,
+            width=width,
+            height=50,
+            title=alt.Title(
+                text="<Drag to select>",
+                dy=30,
+                anchor="middle",
+                fontSize=12,
+                color="gray",
+            ),
+        )
+        .mark_area(interpolate="step-after")
+        .encode(
+            alt.X("key:O", sort="-y", axis=None),
+            alt.Y("n:Q", title=None, axis=None),
+        )
+        .add_params(scrubber_selection)
+    )
+    together = scrubber & zoomin
+    together = together.resolve_scale(color="independent")
+    together = together.properties(
+        title=alt.Title(
+            hist_spec.chart_title,
+            subtitle=[
+                hist_spec.chart_subtitle.format(n_total=n_total),
+                f"{n_keys_total:_} keys total",
+            ],
+            anchor="middle",
+        )
+    )
+    return together