matchlab 0.1.0.dev1__tar.gz → 0.1.0.dev3__tar.gz

{matchlab-0.1.0.dev1 → matchlab-0.1.0.dev3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: matchlab
-Version: 0.1.0.dev1
+Version: 0.1.0.dev3
 Summary: A local-first library for building, running and evaluating entity resolution pipelines.
 Author: Department for Business and Trade
 Project-URL: Documentation, https://uktrade.github.io/matchlab/
@@ -40,19 +40,21 @@ Dynamic: license-file
 Record matching is a chore. matchlab makes it a pipeline you can build, run, query and measure — on your machine, against your warehouse, with nothing to deploy.
 
 ```python
-from matchlab import Source
-from matchlab.models.dedupers import NaiveDeduper
+import matchlab as mb
 
-companies = Source(
-    location=warehouse,
+companies = mb.read_database(
     name="crn",
-    extract_transform="select pk, company, town from companies",
+    sql="select pk, company, town from companies",
+    client=warehouse,
     key_field="pk",
 )
 
 entities = (
     companies.clean({"name": "lower(crn_company)"})
-    .dedupe(model_class=NaiveDeduper, model_settings={"unique_fields": ["name"]})
+    .dedupe(
+        model_class=mb.NaiveDeduper,
+        model_settings={"unique_fields": ["name"]},
+    )
     .resolve()
     .collect()
 )

{matchlab-0.1.0.dev1 → matchlab-0.1.0.dev3}/README.md

@@ -5,19 +5,21 @@
 Record matching is a chore. matchlab makes it a pipeline you can build, run, query and measure — on your machine, against your warehouse, with nothing to deploy.
 
 ```python
-from matchlab import Source
-from matchlab.models.dedupers import NaiveDeduper
+import matchlab as mb
 
-companies = Source(
-    location=warehouse,
+companies = mb.read_database(
     name="crn",
-    extract_transform="select pk, company, town from companies",
+    sql="select pk, company, town from companies",
+    client=warehouse,
     key_field="pk",
 )
 
 entities = (
     companies.clean({"name": "lower(crn_company)"})
-    .dedupe(model_class=NaiveDeduper, model_settings={"unique_fields": ["name"]})
+    .dedupe(
+        model_class=mb.NaiveDeduper,
+        model_settings={"unique_fields": ["name"]},
+    )
     .resolve()
     .collect()
 )

{matchlab-0.1.0.dev1 → matchlab-0.1.0.dev3}/benchmarks/topologies.py

@@ -29,9 +29,7 @@ from typing import cast
 
 from sqlalchemy import create_engine
 
-from matchlab import Model, RecordStep, RelationalDBLocation, Resolver, Source
-from matchlab.models.dedupers import NaiveDeduper
-from matchlab.models.linkers import DeterministicLinker
+import matchlab as mb
 
 # Strip case, whitespace, punctuation and company suffixes, so the cosmetic variation
 # the generator plants is absorbed and matching has something to agree on.
@@ -72,27 +70,25 @@ class Topology(StrEnum):
         }[self]
 
 
-def declare(path: Path, names: Sequence[str]) -> list[Source]:
+def declare(path: Path, names: Sequence[str]) -> list[mb.Source]:
     """Declare one source per table in a generated warehouse.
 
     One engine for all of them, because they are one database and a connection per
     source would measure SQLite's connection handling rather than matchlab's.
     """
-    location = RelationalDBLocation(
-        name="warehouse", client=create_engine(f"sqlite:///{path}")
-    )
+    client = create_engine(f"sqlite:///{path}")
     return [
-        Source(
-            location=location,
-            name=name,
-            extract_transform=f"select key, name, postcode from {name}",
+        mb.read_database(
+            name,
+            sql=f"select key, name, postcode from {name}",
+            client=client,
             key_field="key",
         )
         for name in names
     ]
 
 
-def _cleaning(sources: Sequence[Source]) -> dict[str, str]:
+def _cleaning(sources: Sequence[mb.Source]) -> dict[str, str]:
     """Cleaning expressions that work across however many sources a record step reads.
 
     A multi-source record step concatenates the extracts diagonally, so each source's
@@ -114,8 +110,8 @@ def _cleaning(sources: Sequence[Source]) -> dict[str, str]:
 
 
 def _record_step(
-    sources: Sequence[Source], resolver: Resolver | None = None
-) -> RecordStep:
+    sources: Sequence[mb.Source], resolver: mb.Resolver | None = None
+) -> mb.RecordStep:
     """A cleaned record step of these sources, read through a resolver if given one.
 
     A resolver is itself a record step, read at `id`=entity root, so passing one is the
@@ -124,11 +120,11 @@ def _record_step(
     one is given. Without a resolver, there is a single source, read directly at
     `id`=leaf.
     """
-    base: RecordStep = resolver if resolver is not None else sources[0]
+    base: mb.RecordStep = resolver if resolver is not None else sources[0]
     return base.clean(_cleaning(sources))
 
 
-def _through(resolver: Resolver, source: Source) -> RecordStep:
+def _through(resolver: mb.Resolver, source: mb.Source) -> mb.RecordStep:
     """One source's records, per entity, read through a multi-source resolver.
 
     The resolver's record step carries every source's rows diagonally, so this
@@ -146,23 +142,23 @@ def _through(resolver: Resolver, source: Source) -> RecordStep:
     )
 
 
-def _dedupe(record_step: RecordStep) -> Model:
+def _dedupe(record_step: mb.RecordStep) -> mb.Model:
     """Deduplicate a cleaned record step. Same name and postcode is one entity."""
     return record_step.dedupe(
-        model_class=NaiveDeduper, model_settings={"unique_fields": UNIQUE_FIELDS}
+        model_class=mb.NaiveDeduper, model_settings={"unique_fields": UNIQUE_FIELDS}
     )
 
 
-def _link(left: RecordStep, right: RecordStep) -> Model:
+def _link(left: mb.RecordStep, right: mb.RecordStep) -> mb.Model:
     """Link two cleaned record steps, on the same rule the deduper uses."""
     return left.link(
         right,
-        model_class=DeterministicLinker,
+        model_class=mb.DeterministicLinker,
         model_settings={"comparisons": COMPARISON},
     )
 
 
-def build(topology: Topology, sources: Sequence[Source]) -> Resolver:
+def build(topology: Topology, sources: Sequence[mb.Source]) -> mb.Resolver:
     """Build a plan of this shape over these sources.
 
     Args:
@@ -193,22 +189,24 @@ def build(topology: Topology, sources: Sequence[Source]) -> Resolver:
 
 
 # Which links a star-shaped plan builds, given its record steps.
-Pairs = Callable[[Sequence[RecordStep]], list[tuple[RecordStep, RecordStep]]]
+Pairs = Callable[[Sequence[mb.RecordStep]], list[tuple[mb.RecordStep, mb.RecordStep]]]
 
 
-def _spokes(record_steps: Sequence[RecordStep]) -> list[tuple[RecordStep, RecordStep]]:
+def _spokes(
+    record_steps: Sequence[mb.RecordStep],
+) -> list[tuple[mb.RecordStep, mb.RecordStep]]:
     """Every (hub, spoke) pair. `n-1` links, and the hub in all of them."""
     return [(record_steps[0], spoke) for spoke in record_steps[1:]]
 
 
 def _every_pair(
-    record_steps: Sequence[RecordStep],
-) -> list[tuple[RecordStep, RecordStep]]:
+    record_steps: Sequence[mb.RecordStep],
+) -> list[tuple[mb.RecordStep, mb.RecordStep]]:
     """Every pair of record steps. `n(n-1)/2` links, and no record step privileged."""
     return list(combinations(record_steps, 2))
 
 
-def _star(sources: Sequence[Source], pairs: Pairs) -> Resolver:
+def _star(sources: Sequence[mb.Source], pairs: Pairs) -> mb.Resolver:
     """Dedupe everything into one resolver output, then link the pairs `pairs` chooses.
 
     `HUB` and `MESH` differ only in which pairs get linked, so they share everything
@@ -224,7 +222,7 @@ def _star(sources: Sequence[Source], pairs: Pairs) -> Resolver:
     return links[0].resolve(*links[1:])
 
 
-def _chain(sources: Sequence[Source]) -> Resolver:
+def _chain(sources: Sequence[mb.Source]) -> mb.Resolver:
     """Add one source per level, each level built on the resolver output below it.
 
     Where `MESH` is one resolver output over many links, this is many resolver outputs

matchlab-0.1.0.dev1/docs/api/models.md → matchlab-0.1.0.dev3/docs/api/document.md

@@ -1,4 +1,4 @@
-::: matchlab.models
+::: matchlab.document
     options:
         show_root_heading: true
         show_root_full_path: true

matchlab-0.1.0.dev3/docs/api/locations.md

@@ -0,0 +1,46 @@
+# Locations
+
+The methodologies a [source](sources.md) reads through. A source wraps exactly one location, named by `location_class` and configured by `location_settings`, with its client or frame passed separately as a [resource](resources.md).
+
+::: matchlab.sources.base
+    options:
+        show_root_heading: true
+        show_root_full_path: true
+        members_order: source
+        show_if_no_docstring: true
+        docstring_style: google
+        show_signature_annotations: true
+        separate_signature: true
+        filters:
+            - "!^[A-Z]$"
+            - "!^_"
+
+## Relational databases
+
+::: matchlab.sources.relational
+    options:
+        show_root_heading: true
+        show_root_full_path: true
+        members_order: source
+        show_if_no_docstring: true
+        docstring_style: google
+        show_signature_annotations: true
+        separate_signature: true
+        filters:
+            - "!^[A-Z]$"
+            - "!^_"
+
+## Dataframes
+
+::: matchlab.sources.dataframe
+    options:
+        show_root_heading: true
+        show_root_full_path: true
+        members_order: source
+        show_if_no_docstring: true
+        docstring_style: google
+        show_signature_annotations: true
+        separate_signature: true
+        filters:
+            - "!^[A-Z]$"
+            - "!^_"

matchlab-0.1.0.dev3/docs/api/models.md

@@ -0,0 +1,46 @@
+# Models
+
+The `Model` step, and the methodologies it runs. A model wraps exactly one [deduper](#matchlab.models.dedupers) or [linker](#matchlab.models.linkers), named by `model_class` and configured by `model_settings`.
+
+::: matchlab.models
+    options:
+        show_root_heading: true
+        show_root_full_path: true
+        members_order: source
+        show_if_no_docstring: true
+        docstring_style: google
+        show_signature_annotations: true
+        separate_signature: true
+        filters:
+            - "!^[A-Z]$"
+            - "!^_"
+
+## Dedupers
+
+::: matchlab.models.dedupers
+    options:
+        show_root_heading: true
+        show_root_full_path: true
+        members_order: source
+        show_if_no_docstring: true
+        docstring_style: google
+        show_signature_annotations: true
+        separate_signature: true
+        filters:
+            - "!^[A-Z]$"
+            - "!^_"
+
+## Linkers
+
+::: matchlab.models.linkers
+    options:
+        show_root_heading: true
+        show_root_full_path: true
+        members_order: source
+        show_if_no_docstring: true
+        docstring_style: google
+        show_signature_annotations: true
+        separate_signature: true
+        filters:
+            - "!^[A-Z]$"
+            - "!^_"

matchlab-0.1.0.dev3/docs/api/resources.md

@@ -0,0 +1,12 @@
+::: matchlab.resources
+    options:
+        show_root_heading: true
+        show_root_full_path: true
+        members_order: source
+        show_if_no_docstring: true
+        docstring_style: google
+        show_signature_annotations: true
+        separate_signature: true
+        filters:
+            - "!^[A-Z]$"
+            - "!^_"

{matchlab-0.1.0.dev1 → matchlab-0.1.0.dev3}/docs/api/sources.md

@@ -1,4 +1,8 @@
-::: matchlab.sources
+# Sources
+
+The `Source` step, and the shorthands that build one. A source reads rows through a [location](locations.md) and content-addresses them, so it is where data enters a plan.
+
+::: matchlab.sources.sources
     options:
         show_root_heading: true
         show_root_full_path: true

{matchlab-0.1.0.dev1 → matchlab-0.1.0.dev3}/docs/contributing.md

@@ -47,8 +47,7 @@ The whole suite runs on Python alone. Storage uses DuckDB in memory, and warehou
 just test
 ```
 
-No container is needed, including for the tests that compare SQL dialects. `validate_extract_transform` only asks a client what dialect it speaks, and SQLAlchemy answers that without connecting, so those tests use a Postgres-dialect engine with no driver behind it.
-
+No container is needed: every warehouse test runs against SQLite, over a SQLAlchemy engine or an ADBC connection.
 If you want a real warehouse to point matchlab at while developing, bring up whatever database you like and pass its client to a `Location`. matchlab has no opinion about where it runs, and ships nothing to manage one.
 
 ## Documentation
@@ -88,8 +87,15 @@ When contributing to matchlab and its associated repos, we try to follow consist
 New plan steps subclass [`Step`](api/steps.md). A step must:
 
 * Hold references to its inputs, and to nothing downstream of it
-* Return a stable `_config_key()` covering everything that changes its output, so caching is correct
+* Return a stable `_spec_key()` covering everything that changes its output, so caching is correct
 * Do all its work in `_execute()`, reading inputs from the adapter by fingerprint
+* Settle in `__init__`, listing everything its spec is built from in `_READ_ONLY` so a later assignment raises
+
+That last rule is what keeps a step's configuration, the thing it runs, and its cached artifact from disagreeing. A step builds its methodology from its settings at construction, so an assignment afterwards moves one attribute and nothing derived from it: set `model_class` and the instance still runs the old class, leaving `spec` naming one configuration while `_execute` runs another. The cache leans on the same guarantee from the other side — `_ensure` short-circuits on a stored `_fp` *because* a settled step cannot have changed, so it would hand back the old artifact without consulting the spec at all. Validate first, then write through `_set()`, which is the only thing that gets past the guard.
+
+Derive anything you can rather than storing it. `Transform.transformer_class` and `Model.model_type` are properties over the attributes they describe, so they cannot drift from them.
+
+A methodology must not write into its own settings either, for the same reason: those settings are hashed into the owning step's fingerprint. Copy at the boundary if a library you wrap mutates what you hand it, as `SplinkLinker.prepare` does.
 
 ### Adapters
 

{matchlab-0.1.0.dev1 → matchlab-0.1.0.dev3}/docs/glossary.md

@@ -50,13 +50,23 @@ The 32-byte SHA-256 digest that keys a step's stored [artifact](#artifact). Two
 
 A person's decision that one reviewed cluster's records do, or do not, describe the same [entity](#entity), recorded by matchlab's evaluation tools and scored against a [resolver](#resolver)'s clusters. A judgement is anchored to the content a reviewer was shown, not to any record's key, which is why a [leaf](#leaf) ID, a hash of content, rather than a key, is what a judgement endorses or rejects.
 
+## Key
+
+A record's identifier at the origin, in the [source](#source)'s own data. Its column is named by `key_field`, which defaults to `id`.
+
+A key is not the `id` a [record step](#record-step) carries. That one is a [leaf](#leaf) or a [root](#root), derived by hashing content when the plan is collected. The two never collide, because every column an [extract](#extract) returns is [qualified](#qualify) with the source name first, so a key named `id` reaches a record step as `crn_id`. `leaves()` returns both, under the names `key` and `leaf`.
+
+A key is read as a string whatever the location stores it as, and a null key is refused. `get_lookup()` returns keys, which is how a plan's result joins back to your own tables.
+
+Distinct from an [answer key](#answer-key), which is a [testkit](#testkit) term that shares only the word.
+
 ## Label
 
 A pointer from a name someone chose to a [resolver](#resolver)'s [fingerprint](#fingerprint), created by [publishing](#publish) a resolver's output. A label belongs to the store, not the plan. It can be re-aimed at a different resolver, whereas a source's `name` is part of that source's own output and never moves.
 
 ## Leaf
 
-The stable ID of one record, a hash of its content rather than its key. Identity coming from content rather than key is why a leaf ID changes only when the underlying data does, which is the basis matchlab anchors evaluation judgements to. Reading a source or record step directly, without going through a resolver, exposes the leaf as the `id` column.
+The stable ID of one record, a hash of its content rather than its [key](#key). Identity coming from content rather than key is why a leaf ID changes only when the underlying data does, which is the basis matchlab anchors evaluation judgements to. Reading a source or record step directly, without going through a resolver, exposes the leaf as the `id` column.
 
 ## Linker
 
@@ -64,7 +74,11 @@ A [methodology](#methodology) that finds candidate matches between two [record s
 
 ## Location
 
-matchlab's handle on the warehouse a [source](#source) reads from: a database, plus the client that reaches it. `RelationalDBLocation` is the reference implementation, wrapping a SQLAlchemy or ADBC client. The warehouse is the external database itself; the location is the handle, and it is where a source's rows come *from* — distinct from the [store](#store), where a plan's [artifacts](#artifact) are kept once collected. A location resolves nothing and stores nothing; it only answers a query.
+A generic way of getting a [source](#source)'s rows, and a [methodology](#methodology) like any other. `RelationalDB` runs a query. `DataFrame` reads a frame already in memory. A custom one subclasses `Location` and registers with `add_location_class`.
+
+A location is where a source's rows come from. Do not confuse it with the [store](#store), which keeps a plan's [artifacts](#artifact) once you collect it. A location only reads.
+
+Its [settings](#settings) are hashed into the source's [fingerprint](#fingerprint), so editing a query re-runs the source. Its [resources](#resource) are not hashed, so renaming a warehouse changes nothing.
 
 ## Matcher
 
@@ -78,7 +92,11 @@ The guarantee that a [resolver](#resolver)'s stored output carries forward every
 
 ## Methodology
 
-The matching algorithm a [model](#model) or [resolver](#resolver) step runs. Models run `NaiveDeduper`, `DeterministicLinker`, `SplinkLinker`, or `WeightedDeterministicLinker`. Resolvers run connected components. A model wraps one methodology, chosen by `model_class`, using a [deduper](#deduper) to match within one record step or a [linker](#linker) to match between two. Swapping the methodology only means changing `model_class` (or `resolver_class`), never restructuring the plan around it.
+The pluggable class a [step](#step) runs. Every kind of step has one. A [source](#source) runs a [location](#location), a [transform](#transform) runs a transformer, a [model](#model) runs a [deduper](#deduper) or a [linker](#linker), and a [resolver](#resolver) runs a resolver method.
+
+Every step is built the same way. It names its methodology in a `*_class` argument and configures it with `*_settings`. Anything that cannot be serialised goes in `*_resources`. Swapping a methodology means changing `*_class`, never restructuring the plan around it.
+
+Each kind keeps its own registry, so a custom methodology can be named in a [plan document](#plan-document). Register with `add_location_class`, `add_transformer_class`, `add_model_class`, or `add_resolver_class`.
 
 ## Model
 
@@ -88,13 +106,19 @@ A step that scores candidate matches by running one [methodology](#methodology).
 
 A tree of [steps](#step). Each step holds a reference to its own inputs, so the step you are holding is the pipeline. There is no separate object to register steps with. Nothing runs until you [collect](#collect) it.
 
+## Plan document
+
+A serialisable description of a [plan](#plan), dumped with `dump()` and loaded back with `load()`. Called a **document** for short, which is what the rest of these docs call it. It is JSON, derived from a plan rather than hand-authored, which is why its nodes refer to each other by [position](#position) rather than by any name.
+
+A document carries each step's [settings](#settings) and the edges between steps. It carries no code, no [label](#label), no data, and no [resource](#resource), only the name each resource was given. A plan rebuilt from a document [fingerprints](#fingerprint) identically to the plan it came from, given the same data, so a [store](#store) holding the original's [artifacts](#artifact) serves them rather than redoing the work. `PlanDocument` is the class.
+
 ## Position
 
 Where something falls in an ordered sequence, used instead of a name. The repository uses the word for two different sequences, and the numbers do not agree with each other.
 
 A step's place in `collect()`'s run order is one sense. `draw()` shows it in brackets, and logs quote it (`[step 5]`). Steps have no names, so position is how a log line, a drawn tree, and `plan.lineage()` all refer to the same step. Positions are relative to the step a plan or drawing starts from, so a sub-plan numbers its steps differently from the full plan it came from.
 
-A setting that must point at one of a step's own inputs uses the other sense. It names that input by its index among the step's own inputs, not by that input's place in the whole plan. `ComponentsSettings.thresholds` keys a per-model threshold this way. A model can sit at plan position 5 while still being resolver-input position 0.
+A setting that must point at one of a step's own inputs uses the other sense. It names that input by its index among the step's own inputs, not by that input's place in the whole plan. `Components.thresholds` keys a per-model threshold this way. A model can sit at plan position 5 while still being resolver-input position 0.
 
 ## Prune
 
@@ -108,7 +132,7 @@ Re-publishing the same label at the same resolver output is a no-op. Aiming an e
 
 ## Qualify
 
-Prefix a column name with its [source](#source)'s name (`first_name` becomes `crn_first_name`), so the same field from two sources can sit side by side without colliding. A qualified column's prefix must parse as a valid identifier, which is why a source's `name` is restricted to safe characters.
+Prefix a column name with its [source](#source)'s name (`first_name` becomes `crn_first_name`), so the same field from two sources can sit side by side without colliding. A qualified column's prefix must parse as a valid identifier, which is why a source's `name` is restricted to safe characters, and it is why two different sources in one plan cannot share a name.
 
 ## Record step
 
@@ -124,13 +148,31 @@ A step that collapses a model's scored edges into clusters, one per [entity](#en
 
 Call this a **Resolver**, or its **merge-forwarded Resolver output** if you mean the stored table specifically. Don't call it "a resolution". That noun has been retired.
 
+## Resource
+
+A named object a node needs to run that can't be serialised. Pass one in a step's `*_resources` argument, never among its [settings](#settings). Wrap it as `Resource("warehouse", engine)` to give it the name a [plan document](#plan-document) records. Supply it again when that document is loaded.
+
+For sources, resources are the data-generating mechanism, which affects the [fingerprint](#fingerprint). For all other types of node, resources cannot influence the output of the node because they're ignored by the fingerprint.
+
+A location or methodology marks its resource fields `FromResources`. Every other field is a setting. Passing a field in the wrong argument raises an error that names the field.
+
+Sharing a resource name means sharing a resource. Two steps of different kinds can't share a resource.
+
 ## Root
 
 The ID of a cluster a resolver produces, a hash of the sorted set of [leaf](#leaf) IDs it contains. Two runs that produce the same clustering produce the same root ID, whatever order the underlying algorithm found its clusters in. Reading a [record step](#record-step) through a resolver exposes the root as the `id` column, so several records can share one `id`.
 
+## Settings
+
+A step's serialisable configuration. Passed in a step's `*_settings` argument, carried in a plan document, and hashed into the step's [fingerprint](#fingerprint).
+
+Hashing is what makes settings the opposite of a [resource](#resource): **editing settings re-runs the step and everything below it**, whether or not the output would differ.
+
+Every field of a location or methodology is a setting unless the class marks it a resource.
+
 ## Source
 
-The warehouse query a plan starts from, plus the column that keys it. Every column the query returns is part of a record's identity, so two rows are the same record exactly when the query returns identical values for both.
+The leaf a plan starts from: a [location](#location) to read, plus the column that [keys](#key) it. Every column the location returns is part of a record's identity, so two rows are the same record exactly when it returns identical values for both. Build one with `read_database` or `read_dataframe`.
 
 ## Step