nomenklatura-mpt 4.1.9__tar.gz

nomenklatura_mpt-4.1.9/.gitignore

@@ -0,0 +1,21 @@
+*.pyc
+*.DS_Store
+*.sqlite3
+*.egg-info
+*.perf
+_store/*
+.env
+.mypy_cache
+tests/fixtures/donations.frag.ijson
+tests/fixtures/donations.rslv.ijson
+.coverage
+htmlcov/
+/build/
+/dist/
+/data/
+textual.log
+.nk_cache.db
+nomenklatura.db
+/uv.lock
+/.envrc
+/.vscode

nomenklatura_mpt-4.1.9/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2013-2022, Friedrich Lindenberg
+Copyright (c) 2023-2025, OpenSanctions Datenbanken GmbH
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be included
+in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

nomenklatura_mpt-4.1.9/PKG-INFO

@@ -0,0 +1,159 @@
+Metadata-Version: 2.4
+Name: nomenklatura_mpt
+Version: 4.1.9
+Summary: Make record linkages in followthemoney data.
+Project-URL: Documentation, https://github.com/opensanctions/nomenklatura/
+Project-URL: Repository, https://github.com/opensanctions/nomenklatura.git
+Project-URL: Issues, https://github.com/opensanctions/nomenklatura/issues
+Author-email: OpenSanctions <info@opensanctions.org>
+License: Copyright (c) 2013-2022, Friedrich Lindenberg
+        Copyright (c) 2023-2025, OpenSanctions Datenbanken GmbH
+        
+        Permission is hereby granted, free of charge, to any person obtaining a
+        copy of this software and associated documentation files (the
+        "Software"), to deal in the Software without restriction, including
+        without limitation the rights to use, copy, modify, merge, publish,
+        distribute, sublicense, and/or sell copies of the Software, and to
+        permit persons to whom the Software is furnished to do so, subject to
+        the following conditions:
+        
+        The above copyright notice and this permission notice shall be included
+        in all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+        OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+        MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+        IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+        CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+        TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+        SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+License-File: LICENSE
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Requires-Dist: click
+Requires-Dist: fingerprints
+Requires-Dist: followthemoney
+Requires-Dist: lxml
+Requires-Dist: pydantic
+Requires-Dist: rich
+Requires-Dist: rigour
+Requires-Dist: scikit-learn
+Requires-Dist: shortuuid
+Requires-Dist: sqlalchemy
+Requires-Dist: textual
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: bump2version; extra == 'dev'
+Requires-Dist: coverage>=4.1; extra == 'dev'
+Requires-Dist: fakeredis; extra == 'dev'
+Requires-Dist: flake8>=2.6.0; extra == 'dev'
+Requires-Dist: lxml-stubs; extra == 'dev'
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: plyvel<2.0.0; extra == 'dev'
+Requires-Dist: psycopg2-binary; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: redis<7.0.0,>5.0.0; extra == 'dev'
+Requires-Dist: requests-mock; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Requires-Dist: types-pyyaml; extra == 'dev'
+Requires-Dist: types-redis; extra == 'dev'
+Requires-Dist: types-requests; extra == 'dev'
+Requires-Dist: types-setuptools; extra == 'dev'
+Requires-Dist: wheel>=0.29.0; extra == 'dev'
+Provides-Extra: leveldb
+Requires-Dist: plyvel<2.0.0; extra == 'leveldb'
+Provides-Extra: redis
+Requires-Dist: redis<7.0.0,>5.0.0; extra == 'redis'
+Description-Content-Type: text/markdown
+
+# nomenklatura
+
+Nomenklatura de-duplicates and integrates different [Follow the Money](https://followthemoney.tech/) entities. It serves to clean up messy data and to find links between different datasets.
+
+![screenshot](./docs/screenshot.png)
+
+## Usage
+
+You can install `nomenklatura` via PyPI:
+
+```bash
+$ pip install nomenklatura
+```
+
+### Command-line usage
+
+Much of the functionality of `nomenklatura` can be used as a command-line tool. In the following example, we'll assume that you have a file containing [Follow the Money](https://followthemoney.tech/) entities in your local directory, named `entities.ijson`. If you just want try it out, you can use the file `tests/fixtures/donations.ijson` in this repository for testing (it contains German campaign finance data).
+
+With the file in place, you will cross-reference the entities to generate de-duplication candidates, then run the interactive de-duplication UI in your console, and eventually apply the judgements to generate a new file with merged entities:
+
+```bash
+# generate merge candidates using an in-memory index:
+$ nomenklatura xref entities.ijson
+# note there is now a sqlite database, `nomenklatura.db` that contains de-duplication info.
+$ nomenklatura dedupe entities.ijson
+# will pop up a user interface.
+$ nomenklatura apply entities.ijson -o merged.ijson
+# de-duplicated data goes into `merged.ijson`:
+$ cat entities.ijson | wc -l 
+474
+$ cat merged.ijson | wc -l 
+468 
+```
+
+The resolver graph database location can be customised by setting the environment variable `NOMENKLATURA_DB_URL`
+
+### Programmatic usage
+
+The command-line use of `nomenklatura` is targeted at small datasets which need to be de-duplicated. For more involved scenarios, the package also offers a Python API which can be used to control the semantics of de-duplication.
+
+* `nomenklatura.Dataset` - implements a basic dataset for describing a set of entities.
+* `nomenklatura.Store` - a general purpose access mechanism for entities. By default, a store is used to access entity data stored in files as an in-memory cache, but the store can be subclassed to work with entities from a database system.
+* `nomenklatura.Index` - a full-text in-memory search index for FtM entities. In the application, this is used to block de-duplication candidates, but the index can also be used to drive an API etc.
+* `nomenklatura.index.TantivyIndex` - a wrapper around Tantivy for indexing and matching FtM entities.
+* `nomenklatura.Resolver` - the core of the de-duplication process, the resolver is essentially a graph with edges made out of entity judgements. The resolver can be used to store judgements or get the canonical ID for a given entity.
+
+All of the API classes have extensive type annotations, which should make their integration in any modern Python API simpler.
+
+## Design
+
+This package offers an implementation of an in-memory data deduplication framework centered around the FtM data model. The idea is the following workflow:
+
+* Accept FtM-shaped entities from a given loader (e.g. a JSON file, or a database)
+* Build an in-memory inverted index of the entities for dedupe blocking
+* Generate merge candidates using the blocking index and FtM compare
+* Provide a file-based storage format for merge challenges and decisions
+* Provide a text-based user interface to let users make merge decisions
+
+Later on, the following might be added:
+
+* A web application to let users make merge decisions on the web
+
+### Resolver graph
+
+The key implementation detail of nomenklatura is the `Resolver`, a graph structure that
+manages user decisions regarding entity identity. Edges are `Judgements` of whether
+two entity IDs are the same, not the same, or undecided. The resolver implements an
+algorithm for computing connected components, which can the be used to find the best
+available ID for a cluster of entities. It can also be used to evaluate transitive
+judgements, e.g. if A <> B, and B = C, then we don't need to ask if A = C.
+
+## Reading
+
+* https://dedupe.readthedocs.org/en/latest/
+* https://github.com/OpenRefine/OpenRefine/wiki/Reconcilable-Data-Sources
+* https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth
+* https://github.com/OpenRefine/OpenRefine/wiki/Reconciliation-Service-API
+
+
+## Contact, contributions etc.
+
+This codebase is licensed under the terms of an MIT license (see LICENSE).
+
+We're keen for any contributions, bug fixes and feature suggestions, please use the GitHub issue tracker for this repository. 
+
+Nomenklatura is currently developed thanks to a Prototypefund grant for [OpenSanctions](https://opensanctions.org). Previous iterations of the package were developed with support from [Knight-Mozilla OpenNews](http://opennews.org) and the [Open Knowledge Foundation Labs](http://okfnlabs.org).

nomenklatura_mpt-4.1.9/README.md

@@ -0,0 +1,86 @@
+# nomenklatura
+
+Nomenklatura de-duplicates and integrates different [Follow the Money](https://followthemoney.tech/) entities. It serves to clean up messy data and to find links between different datasets.
+
+![screenshot](./docs/screenshot.png)
+
+## Usage
+
+You can install `nomenklatura` via PyPI:
+
+```bash
+$ pip install nomenklatura
+```
+
+### Command-line usage
+
+Much of the functionality of `nomenklatura` can be used as a command-line tool. In the following example, we'll assume that you have a file containing [Follow the Money](https://followthemoney.tech/) entities in your local directory, named `entities.ijson`. If you just want try it out, you can use the file `tests/fixtures/donations.ijson` in this repository for testing (it contains German campaign finance data).
+
+With the file in place, you will cross-reference the entities to generate de-duplication candidates, then run the interactive de-duplication UI in your console, and eventually apply the judgements to generate a new file with merged entities:
+
+```bash
+# generate merge candidates using an in-memory index:
+$ nomenklatura xref entities.ijson
+# note there is now a sqlite database, `nomenklatura.db` that contains de-duplication info.
+$ nomenklatura dedupe entities.ijson
+# will pop up a user interface.
+$ nomenklatura apply entities.ijson -o merged.ijson
+# de-duplicated data goes into `merged.ijson`:
+$ cat entities.ijson | wc -l 
+474
+$ cat merged.ijson | wc -l 
+468 
+```
+
+The resolver graph database location can be customised by setting the environment variable `NOMENKLATURA_DB_URL`
+
+### Programmatic usage
+
+The command-line use of `nomenklatura` is targeted at small datasets which need to be de-duplicated. For more involved scenarios, the package also offers a Python API which can be used to control the semantics of de-duplication.
+
+* `nomenklatura.Dataset` - implements a basic dataset for describing a set of entities.
+* `nomenklatura.Store` - a general purpose access mechanism for entities. By default, a store is used to access entity data stored in files as an in-memory cache, but the store can be subclassed to work with entities from a database system.
+* `nomenklatura.Index` - a full-text in-memory search index for FtM entities. In the application, this is used to block de-duplication candidates, but the index can also be used to drive an API etc.
+* `nomenklatura.index.TantivyIndex` - a wrapper around Tantivy for indexing and matching FtM entities.
+* `nomenklatura.Resolver` - the core of the de-duplication process, the resolver is essentially a graph with edges made out of entity judgements. The resolver can be used to store judgements or get the canonical ID for a given entity.
+
+All of the API classes have extensive type annotations, which should make their integration in any modern Python API simpler.
+
+## Design
+
+This package offers an implementation of an in-memory data deduplication framework centered around the FtM data model. The idea is the following workflow:
+
+* Accept FtM-shaped entities from a given loader (e.g. a JSON file, or a database)
+* Build an in-memory inverted index of the entities for dedupe blocking
+* Generate merge candidates using the blocking index and FtM compare
+* Provide a file-based storage format for merge challenges and decisions
+* Provide a text-based user interface to let users make merge decisions
+
+Later on, the following might be added:
+
+* A web application to let users make merge decisions on the web
+
+### Resolver graph
+
+The key implementation detail of nomenklatura is the `Resolver`, a graph structure that
+manages user decisions regarding entity identity. Edges are `Judgements` of whether
+two entity IDs are the same, not the same, or undecided. The resolver implements an
+algorithm for computing connected components, which can the be used to find the best
+available ID for a cluster of entities. It can also be used to evaluate transitive
+judgements, e.g. if A <> B, and B = C, then we don't need to ask if A = C.
+
+## Reading
+
+* https://dedupe.readthedocs.org/en/latest/
+* https://github.com/OpenRefine/OpenRefine/wiki/Reconcilable-Data-Sources
+* https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth
+* https://github.com/OpenRefine/OpenRefine/wiki/Reconciliation-Service-API
+
+
+## Contact, contributions etc.
+
+This codebase is licensed under the terms of an MIT license (see LICENSE).
+
+We're keen for any contributions, bug fixes and feature suggestions, please use the GitHub issue tracker for this repository. 
+
+Nomenklatura is currently developed thanks to a Prototypefund grant for [OpenSanctions](https://opensanctions.org). Previous iterations of the package were developed with support from [Knight-Mozilla OpenNews](http://opennews.org) and the [Open Knowledge Foundation Labs](http://okfnlabs.org).

nomenklatura_mpt-4.1.9/nomenklatura/__init__.py

@@ -0,0 +1,11 @@
+from nomenklatura.resolver import Resolver
+from nomenklatura.store import Store, View
+from nomenklatura.index import Index
+
+__version__ = "4.1.9"
+__all__ = [
+    "Resolver",
+    "Index",
+    "Store",
+    "View",
+]

nomenklatura_mpt-4.1.9/nomenklatura/cache.py

@@ -0,0 +1,194 @@
+import math
+import json
+import logging
+from random import randint
+from dataclasses import dataclass
+from typing import Any, cast, Dict, Optional, Union, Generator
+from datetime import datetime, timedelta
+from sqlalchemy import MetaData
+from sqlalchemy import Table, Column, DateTime, Unicode
+from sqlalchemy.engine import Engine, Connection, Transaction
+from sqlalchemy.future import select
+from sqlalchemy.sql.expression import delete
+from sqlalchemy.exc import OperationalError, InvalidRequestError
+from sqlalchemy.dialects.postgresql import insert as upsert
+from rigour.time import naive_now
+from followthemoney import Dataset
+
+from nomenklatura.db import get_engine, get_metadata
+
+
+log = logging.getLogger(__name__)
+Value = Union[str, None]
+
+
+@dataclass
+class CacheValue:
+    key: str
+    dataset: Optional[str]
+    text: Value
+    timestamp: datetime
+
+
+def randomize_cache(days: int) -> timedelta:
+    min_cache = max(1, math.ceil(days * 0.5))
+    max_cache = math.ceil(days * 1.3)
+    return timedelta(days=randint(min_cache, max_cache))
+
+
+class Cache(object):
+    def __init__(
+        self, engine: Engine, metadata: MetaData, dataset: Dataset, create: bool = False
+    ) -> None:
+        self.dataset = dataset
+        self._engine = engine
+        self._conn: Optional[Connection] = None
+        self._transaction: Optional[Transaction] = None
+        self._table = Table(
+            "cache",
+            metadata,
+            Column("key", Unicode(), primary_key=True),
+            Column("text", Unicode(), nullable=True),
+            Column("dataset", Unicode(), nullable=False),
+            Column("timestamp", DateTime, index=True),
+            extend_existing=True,
+        )
+        if create:
+            metadata.create_all(bind=engine, checkfirst=True, tables=[self._table])
+
+        self._preload: Dict[str, CacheValue] = {}
+
+    @property
+    def conn(self) -> Connection:
+        if self._conn is None:
+            self._conn = self._engine.connect()
+            self._transaction = self._conn.begin()
+        return self._conn
+
+    def set(self, key: str, value: Value) -> None:
+        self._preload.pop(key, None)
+        cache = {
+            "timestamp": naive_now(),
+            "key": key,
+            "dataset": self.dataset.name,
+            "text": value,
+        }
+        try:
+            istmt = upsert(self._table).values(cache)
+            values = dict(
+                timestamp=istmt.excluded.timestamp,
+                text=istmt.excluded.text,
+                dataset=istmt.excluded.dataset,
+            )
+            stmt = istmt.on_conflict_do_update(index_elements=["key"], set_=values)
+            self.conn.execute(stmt)
+        except (OperationalError, InvalidRequestError) as exc:
+            log.exception("Error while saving to cache: %s" % exc)
+            self.reset()
+
+    def set_json(self, key: str, value: Any) -> None:
+        return self.set(key, json.dumps(value))
+
+    def get(self, key: str, max_age: Optional[int] = None) -> Optional[Value]:
+        if max_age is not None and max_age < 1:
+            return None
+
+        cache_cutoff = None
+        if max_age is not None:
+            cache_cutoff = naive_now() - randomize_cache(max_age)
+
+        cache = self._preload.get(key)
+        if cache is not None:
+            if cache_cutoff is not None and cache.timestamp < cache_cutoff:
+                return None
+            return cache.text
+
+        q = select(self._table.c.text)
+        q = q.filter(self._table.c.key == key)
+        if cache_cutoff is not None:
+            q = q.filter(self._table.c.timestamp > cache_cutoff)
+        q = q.order_by(self._table.c.timestamp.desc())
+        q = q.limit(1)
+        try:
+            result = self.conn.execute(q)
+            row = result.fetchone()
+        except InvalidRequestError as ire:
+            log.exception("Cache fetch error: %s", ire)
+            self.reset()
+            return None
+        if row is not None:
+            return cast(Optional[str], row.text)
+        return None
+
+    def get_json(self, key: str, max_age: Optional[int] = None) -> Optional[Any]:
+        text = self.get(key, max_age=max_age)
+        if text is None:
+            return None
+        return json.loads(text)
+
+    def has(self, key: str) -> bool:
+        return self.get(key) is not None
+
+    def delete(self, key: str) -> None:
+        self._preload.pop(key, None)
+        pq = delete(self._table)
+        pq = pq.where(self._table.c.key == key)
+        try:
+            self.conn.execute(pq)
+        except InvalidRequestError as ire:
+            log.exception("Cache delete error: %s", ire)
+            self.reset()
+            return None
+
+    def all(self, like: Optional[str]) -> Generator[CacheValue, None, None]:
+        q = select(self._table)
+        if like is not None:
+            q = q.filter(self._table.c.key.like(like))
+
+        result = self.conn.execute(q)
+        for row in result.yield_per(10000):
+            yield CacheValue(row.key, row.dataset, row.text, row.timestamp)
+
+    def preload(self, like: Optional[str] = None) -> None:
+        log.info("Pre-loading cache: %r", like)
+        for cache in self.all(like=like):
+            self._preload[cache.key] = cache
+
+    def clear(self) -> None:
+        try:
+            pq = delete(self._table)
+            pq = pq.where(self._table.c.dataset == self.dataset.name)
+            self.conn.execute(pq)
+        except InvalidRequestError:
+            log.exception("Cannot clear cache from database")
+            self.reset()
+
+    def reset(self) -> None:
+        if self._conn is not None:
+            self._conn.close()
+        self._conn = None
+        self._transaction = None
+
+    def flush(self) -> None:
+        # log.info("Flushing cache.")
+        if self._transaction is not None:
+            try:
+                self._transaction.commit()
+            except InvalidRequestError:
+                log.exception("Transaction was failed, cannot store cache state.")
+        self.reset()
+
+    def close(self) -> None:
+        self.flush()
+
+    def __repr__(self) -> str:
+        return f"<Cache({self._table!r})>"
+
+    def __hash__(self) -> int:
+        return hash((self.dataset.name, self._table.name))
+
+    @classmethod
+    def make_default(cls, dataset: Dataset) -> "Cache":
+        engine = get_engine()
+        metadata = get_metadata()
+        return cls(engine, metadata, dataset, create=True)