reckoner-er 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- reckoner_er-0.3.0/CHANGELOG.md +37 -0
- reckoner_er-0.3.0/LICENSE +21 -0
- reckoner_er-0.3.0/MANIFEST.in +8 -0
- reckoner_er-0.3.0/PKG-INFO +100 -0
- reckoner_er-0.3.0/README.md +268 -0
- reckoner_er-0.3.0/benchmarks/data/gleif_us_sample.csv +2923 -0
- reckoner_er-0.3.0/benchmarks/data/gleif_us_sample.meta.json +50 -0
- reckoner_er-0.3.0/benchmarks/fetch_gleif_sample.py +162 -0
- reckoner_er-0.3.0/benchmarks/run_benchmark.py +142 -0
- reckoner_er-0.3.0/docs/benchmark.md +91 -0
- reckoner_er-0.3.0/docs/pypi.md +70 -0
- reckoner_er-0.3.0/docs/python-api.md +109 -0
- reckoner_er-0.3.0/examples/api.py +31 -0
- reckoner_er-0.3.0/examples/quickstart.py +34 -0
- reckoner_er-0.3.0/examples/vendors.csv +10 -0
- reckoner_er-0.3.0/pyproject.toml +63 -0
- reckoner_er-0.3.0/reckoner/__init__.py +77 -0
- reckoner_er-0.3.0/reckoner/__main__.py +7 -0
- reckoner_er-0.3.0/reckoner/api.py +221 -0
- reckoner_er-0.3.0/reckoner/cli.py +474 -0
- reckoner_er-0.3.0/reckoner/evaluate.py +89 -0
- reckoner_er-0.3.0/reckoner/py.typed +0 -0
- reckoner_er-0.3.0/reckoner/resolver.py +341 -0
- reckoner_er-0.3.0/reckoner_er.egg-info/PKG-INFO +100 -0
- reckoner_er-0.3.0/reckoner_er.egg-info/SOURCES.txt +30 -0
- reckoner_er-0.3.0/reckoner_er.egg-info/dependency_links.txt +1 -0
- reckoner_er-0.3.0/reckoner_er.egg-info/entry_points.txt +2 -0
- reckoner_er-0.3.0/reckoner_er.egg-info/top_level.txt +1 -0
- reckoner_er-0.3.0/setup.cfg +4 -0
- reckoner_er-0.3.0/tests/test_api.py +299 -0
- reckoner_er-0.3.0/tests/test_cli.py +127 -0
- reckoner_er-0.3.0/tests/test_resolver.py +89 -0
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.3.0] - 2026-11-14
|
|
4
|
+
|
|
5
|
+
- On PyPI as `reckoner-er` (`pip install reckoner-er`). The name `reckoner` there belongs to an unrelated Helm tool; the import name and the command stay `reckoner`.
|
|
6
|
+
- New two-function API: `resolve(names)` returns typed `Cluster` objects (members, names, canonical name, identifiers, confidence, and an `explanations` sentence for every merge and refusal); `match(a, b)` returns a `MatchResult` with `matched`, `score`, `method` and `reasons`. Both take plain strings or records with identifiers. `Cluster` and `MatchResult` are frozen and hashable. See `docs/python-api.md`.
|
|
7
|
+
- New `reckoner match "A" "B"` command, with `--a-cik`/`--b-cik` (and UEI, EIN, CAGE, ticker, domain), `--agency`, `--color` and `--json`. Exits 0 on a match and 1 otherwise. `match` is recognised as the first argument that is not an option, so `reckoner --color never match A B` works too. Because of this, a file literally named `match` is now read with `reckoner ./match` or `reckoner -- match`.
|
|
8
|
+
- `pairwise_scores(clusters, labels)` scores a resolution against labelled data (pairwise precision, recall, F1).
|
|
9
|
+
- Benchmark on real GLEIF names (1312 entities, 2922 names, CC0 snapshot committed): precision 0.966 and recall 0.969 on formatting variants; renames and trade names are not linked, by design. `python benchmarks/run_benchmark.py`; details in `docs/benchmark.md`. CI fails if precision or recall drops below 0.95.
|
|
10
|
+
- Fix: dotted legal suffixes at the end of a name are now stripped like the plain ones. "Oracle Partners, L.P." and "ORACLE PARTNERS LP" both normalize to `oracle partners` (they did not match before), as do "L.L.C.", "P.A." and similar. Each dotted abbreviation is read on its own from the right, so "Acme U.S. L.P." matches "Acme U.S. LP" and "Foo L.P. L.L.C." matches "Foo LP LLC". Only letters that spell a legal suffix are joined, so initials such as "A.G. Edwards" and "U.S." normalize exactly as before. On the benchmark this raised formatting recall from 0.910 to 0.969 with precision unchanged.
|
|
11
|
+
- Type hints throughout; ships `py.typed` and passes `mypy --strict`.
|
|
12
|
+
- `publish.yml` builds, checks and smoke-tests the sdist and wheel, then uploads them to PyPI with trusted publishing when a `v*` tag is pushed.
|
|
13
|
+
- The PyPI page comes from `docs/pypi.md` (absolute image URLs, `pip install reckoner-er`). The sdist includes `benchmarks/`, `docs/` and `examples/`, so the full test suite runs from an unpacked sdist; the benchmark test skips itself when `benchmarks/` is absent.
|
|
14
|
+
|
|
15
|
+
## [0.2.2] - 2026-09-25
|
|
16
|
+
|
|
17
|
+
- Fix: `link_confidence` no longer claims an identifier's confidence when only one record in the cluster carries that identifier. Such clusters were joined by name, so they now report 0.60 (for example GE with a CIK on one record only: was 0.99, now 0.60).
|
|
18
|
+
- The summary now shows every record in each group and why it joined (`= CIK 907471`, `= name 'general electric'`), aligned, with a count of merges and refusals.
|
|
19
|
+
- Color in the terminal (off when piped, with `NO_COLOR`, or `--color never`).
|
|
20
|
+
- Running `reckoner` with no file prints what to try next; a file with none of the recognised columns stops with a message naming the columns found and what to rename; a missing file suggests `--demo`.
|
|
21
|
+
- `--help` ends with copy-paste examples.
|
|
22
|
+
- `install.sh` and `install.ps1` one-line installers; `examples/vendors.csv`.
|
|
23
|
+
|
|
24
|
+
## [0.2.1] - 2026-09-25
|
|
25
|
+
|
|
26
|
+
- The CLI now stops with a clear message when a CSV row has more fields than the header (usually an unquoted comma in a company name) instead of writing a garbled extra column.
|
|
27
|
+
|
|
28
|
+
## [0.2.0] - 2026-09-25
|
|
29
|
+
|
|
30
|
+
- New `reckoner` command line tool: resolve a CSV, JSON or JSON Lines file (or stdin) and get a readable summary, the full JSON result with every receipt, or a CSV of your rows with `entity_id`, `canonical_name` and `link_confidence` appended. `--agency` for agency names, `--demo` for a built-in example.
|
|
31
|
+
- Prebuilt single-file executables for Windows, macOS (Apple Silicon and Intel) and Linux on every GitHub Release. No Python needed.
|
|
32
|
+
- `python -m reckoner` works too.
|
|
33
|
+
- CI runs the tests and the CLI on Linux and Windows.
|
|
34
|
+
|
|
35
|
+
## [0.1.0]
|
|
36
|
+
|
|
37
|
+
- Initial release: `EntityResolver`, identifier-first merges, exact-normalized name links, refusal receipts, agency mode.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Matthew Busel
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# The sdist carries what the test suite needs: tests/test_api.py runs the
|
|
2
|
+
# benchmark on the committed GLEIF snapshot.
|
|
3
|
+
recursive-include tests *.py
|
|
4
|
+
recursive-include benchmarks *.py *.csv *.json
|
|
5
|
+
recursive-include docs *.md
|
|
6
|
+
recursive-include examples *.py *.csv
|
|
7
|
+
include CHANGELOG.md
|
|
8
|
+
global-exclude __pycache__ *.py[cod]
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: reckoner-er
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Summary: Deterministic, explainable entity resolution for organization names. No fuzzy matching. Every merge comes with a receipt.
|
|
5
|
+
Author-email: Matthew Busel <mattbusel@gmail.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/Mattbusel/reckoner
|
|
8
|
+
Project-URL: Issues, https://github.com/Mattbusel/reckoner/issues
|
|
9
|
+
Project-URL: Changelog, https://github.com/Mattbusel/reckoner/blob/main/CHANGELOG.md
|
|
10
|
+
Project-URL: Documentation, https://github.com/Mattbusel/reckoner/blob/main/docs/python-api.md
|
|
11
|
+
Keywords: entity-resolution,record-linkage,deduplication,company-names,data-cleaning,data-engineering,master-data-management,name-matching,lei
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Environment :: Console
|
|
15
|
+
Classifier: Operating System :: OS Independent
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
23
|
+
Classifier: Typing :: Typed
|
|
24
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
25
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
26
|
+
Requires-Python: >=3.9
|
|
27
|
+
Description-Content-Type: text/markdown
|
|
28
|
+
License-File: LICENSE
|
|
29
|
+
Dynamic: license-file
|
|
30
|
+
|
|
31
|
+
<p align="center"><img src="https://raw.githubusercontent.com/Mattbusel/reckoner/v0.3.0/assets/banner.png" alt="reckoner: input records on the left, joined to the companies they name on the right, each line labeled with the reason; one pair stamped REFUSED" width="100%"></p>
|
|
32
|
+
|
|
33
|
+
# reckoner
|
|
34
|
+
|
|
35
|
+
**Give it a list of company names from different sources, and it tells you which ones are the same company, and why.** It only merges on proof (a shared ID, or the exact same name once spelling noise is removed), never on a guess, and it writes down a receipt for every merge and every refusal.
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
pip install reckoner-er
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
The package is called `reckoner-er` on PyPI because `reckoner` there is an unrelated Helm tool, so `pip install reckoner` installs the wrong thing. The import name and the command are both `reckoner`. Do not install both packages in one environment. Python 3.9 or newer, no dependencies, ships `py.typed`.
|
|
42
|
+
|
|
43
|
+
<p align="center"><img src="https://raw.githubusercontent.com/Mattbusel/reckoner/v0.3.0/assets/demo.gif" alt="Terminal recording: reckoner --demo, then reckoner vendors.csv grouping 9 rows into 5 companies, then writing resolved.csv with an entity_id column" width="900"></p>
|
|
44
|
+
|
|
45
|
+
## From Python
|
|
46
|
+
|
|
47
|
+
```python
|
|
48
|
+
from reckoner import resolve, match
|
|
49
|
+
|
|
50
|
+
clusters = resolve([
|
|
51
|
+
"GENERAL ELECTRIC CO",
|
|
52
|
+
"General Electric Company",
|
|
53
|
+
{"name": "GE", "cik": "0000040545"},
|
|
54
|
+
{"name": "General Electric", "cik": "40545"},
|
|
55
|
+
"Boeing",
|
|
56
|
+
])
|
|
57
|
+
for c in clusters:
|
|
58
|
+
print(c.canonical_name, c.members, c.link_confidence)
|
|
59
|
+
# General Electric Company (0, 1, 2, 3) 0.99
|
|
60
|
+
# Boeing (4,) 1.0
|
|
61
|
+
|
|
62
|
+
m = match("THE SHERWIN-WILLIAMS COMPANY", "Sherwin Williams Co")
|
|
63
|
+
m.matched, m.score, m.method # (True, 0.6, 'name_exact_normalized')
|
|
64
|
+
|
|
65
|
+
match({"name": "Meta Platforms, Inc.", "cik": "1326801"},
|
|
66
|
+
{"name": "META PLATFORMS INC", "cik": "907471"}).reasons
|
|
67
|
+
# ('identical normalized names but conflicting CIK - refusing to merge',)
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Every `Cluster` carries its members, identifiers, confidence and an `explanations` sentence for every merge and refusal. Full reference: [docs/python-api.md](https://github.com/Mattbusel/reckoner/blob/v0.3.0/docs/python-api.md).
|
|
71
|
+
|
|
72
|
+
## From the command line
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
reckoner --demo # see it work on a built-in example
|
|
76
|
+
reckoner vendors.csv # CSV, JSON or JSON Lines with a name column
|
|
77
|
+
reckoner vendors.csv -f csv -o resolved.csv # your rows + entity_id, canonical_name, link_confidence
|
|
78
|
+
reckoner match "GENERAL ELECTRIC CO" "General Electric Company" # exit 0 on a match, 1 otherwise
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Recognised columns (case-insensitive): `name`, `agency`, `cik`, `uei`, `ein`, `cage`, `ticker`, `domain`, `source`, `state`. `--agency` handles government agency names (`DoD`, `EPA`, `U.S.` prefixes). `reckoner --help` lists every option.
|
|
82
|
+
|
|
83
|
+
## How it decides
|
|
84
|
+
|
|
85
|
+
<p align="center"><img src="https://raw.githubusercontent.com/Mattbusel/reckoner/v0.3.0/assets/how-it-decides.svg" alt="How reckoner decides: pass 1 merges on a shared ID; pass 2 links exact normalized names at 0.60 unless the records carry conflicting IDs, in which case it refuses with a reason" width="880"></p>
|
|
86
|
+
|
|
87
|
+
- **Identifiers merge first.** A shared CIK, UEI, EIN, CAGE, ticker or domain merges two records outright. Registry IDs (CIK, UEI, EIN, CAGE) link at 0.97 to 0.99; reassignable ones (ticker 0.90, domain 0.85) lower.
|
|
88
|
+
- **Names link only on an exact match after conservative cleanup.** Case, punctuation, `&`, a leading `The`, and trailing legal suffixes (`Inc`, `Corp`, `LLC`, `L.P.`, `GmbH`, `plc`, ...) are removed. Distinguishing words are never dropped. Such links score 0.60.
|
|
89
|
+
- **No fuzzy scoring anywhere.** A typo does not match. There is no similarity knob to tune.
|
|
90
|
+
- **Conflicting registry IDs block a merge.** Same normalized name but different CIKs: refused, with the reason recorded.
|
|
91
|
+
|
|
92
|
+
On a committed snapshot of real GLEIF legal names (1312 entities, 2922 names), formatting variants resolve at precision 0.966 and recall 0.969. Renames and trade names are not linked, by design. Method and numbers: [docs/benchmark.md](https://github.com/Mattbusel/reckoner/blob/v0.3.0/docs/benchmark.md).
|
|
93
|
+
|
|
94
|
+
## Links
|
|
95
|
+
|
|
96
|
+
- Source, install scripts and prebuilt binaries (no Python needed): https://github.com/Mattbusel/reckoner
|
|
97
|
+
- Changelog: https://github.com/Mattbusel/reckoner/blob/main/CHANGELOG.md
|
|
98
|
+
- Issues: https://github.com/Mattbusel/reckoner/issues
|
|
99
|
+
|
|
100
|
+
MIT licensed.
|
|
@@ -0,0 +1,268 @@
|
|
|
1
|
+
<p align="center"><img src="assets/banner.png" alt="reckoner: input records on the left, joined to the companies they name on the right, each line labeled with the reason; one pair stamped REFUSED" width="100%"></p>
|
|
2
|
+
|
|
3
|
+
# reckoner
|
|
4
|
+
|
|
5
|
+
**Give it a spreadsheet of company names from different sources, and it tells you which rows are the same company, and why.** It only merges on proof (a shared ID, or the exact same name once spelling noise is removed), never on a guess, and it writes down a receipt for every merge and every refusal.
|
|
6
|
+
|
|
7
|
+
<p align="center"><img src="assets/demo.gif" alt="Terminal recording: reckoner --demo, then reckoner vendors.csv grouping 9 rows into 5 companies, then writing resolved.csv with an entity_id column" width="900"></p>
|
|
8
|
+
|
|
9
|
+
## Install
|
|
10
|
+
|
|
11
|
+
| Your system | One line |
|
|
12
|
+
| --- | --- |
|
|
13
|
+
| Windows (PowerShell) | `irm https://raw.githubusercontent.com/Mattbusel/reckoner/main/install.ps1 \| iex` |
|
|
14
|
+
| macOS or Linux | `curl -fsSL https://raw.githubusercontent.com/Mattbusel/reckoner/main/install.sh \| sh` |
|
|
15
|
+
| Homebrew (macOS, Linux) | `brew install mattbusel/tap/reckoner` |
|
|
16
|
+
| Scoop (Windows) | `scoop bucket add mattbusel https://github.com/Mattbusel/scoop-bucket; scoop install mattbusel/reckoner` |
|
|
17
|
+
| Python (pipx) | `pipx install git+https://github.com/Mattbusel/reckoner` |
|
|
18
|
+
| Just the file | [Download for Windows, macOS or Linux](https://github.com/Mattbusel/reckoner/releases/latest), unzip, run |
|
|
19
|
+
|
|
20
|
+
No Python is needed for anything except the pipx line. The install scripts check the download against the release's `SHA256SUMS.txt` before installing: `install.sh` puts `reckoner` in `~/.local/bin`, `install.ps1` puts `reckoner.exe` in `%LOCALAPPDATA%\Programs\reckoner` and adds it to your PATH.
|
|
21
|
+
|
|
22
|
+
<details>
|
|
23
|
+
<summary>Unsigned binary warnings, and why not <code>pip install reckoner</code></summary>
|
|
24
|
+
|
|
25
|
+
The binaries are unsigned. Windows SmartScreen may say "unknown publisher": click **More info**, then **Run anyway**. On macOS, if a downloaded binary is blocked, run `xattr -d com.apple.quarantine reckoner` (the install script does this for you).
|
|
26
|
+
|
|
27
|
+
The name `reckoner` on PyPI belongs to an unrelated Helm tool, so `pip install reckoner` installs the wrong thing. Use the pipx line above, or `pip install git+https://github.com/Mattbusel/reckoner` for the Python library.
|
|
28
|
+
</details>
|
|
29
|
+
|
|
30
|
+
## Use it in 3 steps
|
|
31
|
+
|
|
32
|
+
**1. See it work on built-in data.**
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
reckoner --demo
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
You get 7 records grouped into 4 companies, and one refused merge: two records that are both called "Meta Platforms" but carry different SEC CIK numbers, so reckoner will not join them.
|
|
39
|
+
|
|
40
|
+
**2. Point it at your own file.** A CSV, JSON or JSON Lines file with a `name` column (and, if you have them, ID columns like `cik`, `ein`, `uei`, `ticker`, `domain`):
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
reckoner vendors.csv
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Every group is listed with the rows in it and the reason each row joined.
|
|
47
|
+
|
|
48
|
+
**3. Save the answer as a spreadsheet.**
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
reckoner vendors.csv -f csv -o resolved.csv
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
`resolved.csv` is your file, same rows, same order, with three columns added: `entity_id` (rows with the same ID are the same company), `canonical_name` and `link_confidence`.
|
|
55
|
+
|
|
56
|
+
## Results
|
|
57
|
+
|
|
58
|
+
This is the real output for [`examples/vendors.csv`](examples/vendors.csv), nine rows from three made-up feeds:
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
$ reckoner examples/vendors.csv
|
|
62
|
+
9 records -> 5 entities 4 merges, 0 refused
|
|
63
|
+
|
|
64
|
+
ent-0 General Electric Company 0.60 TICKER GE
|
|
65
|
+
#0 GENERAL ELECTRIC CO sec
|
|
66
|
+
#1 General Electric Company usaspending = name 'general electric'
|
|
67
|
+
ent-2 The Boeing Company 0.60 TICKER BA
|
|
68
|
+
#2 The Boeing Company sec
|
|
69
|
+
#3 BOEING CO usaspending = name 'boeing'
|
|
70
|
+
ent-4 Lockheed Martin Corporation 0.90 TICKER LMT
|
|
71
|
+
#4 Lockheed Martin Corporation sec
|
|
72
|
+
#5 LOCKHEED MARTIN CORP usaspending = name 'lockheed martin'
|
|
73
|
+
#6 Lockheed Martin Corp. sam = TICKER LMT
|
|
74
|
+
ent-7 Microsoft Corporation 1.00 TICKER MSFT
|
|
75
|
+
ent-8 Micrsoft Corp 1.00
|
|
76
|
+
|
|
77
|
+
#n is the input record. '= CIK 123', '= TICKER X': joined on a shared ID.
|
|
78
|
+
'= name ...' joined on the exact normalized name (0.60, never fuzzy).
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Read it like this:
|
|
82
|
+
|
|
83
|
+
- **Lockheed Martin**: row 6 joined because it has the same ticker as row 4 (0.90). Row 5 has no ticker, but its name is the same once case and "CORP" are removed, so it links at 0.60.
|
|
84
|
+
- **Boeing**: "The Boeing Company" and "BOEING CO" both become `boeing`.
|
|
85
|
+
- **Micrsoft Corp stays on its own.** It is a typo, and reckoner does not guess. A wrong merge is worse than a missed one.
|
|
86
|
+
|
|
87
|
+
And when two rows look like the same name but carry different registry IDs, the merge is refused and the reason is kept (from `reckoner --demo`):
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
Refused merges (1):
|
|
91
|
+
#4 + #6 'Meta Platforms, Inc.' vs 'META PLATFORMS INC'
|
|
92
|
+
identical normalized names but conflicting CIK - refusing to merge
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
<p align="center"><img src="assets/how-it-decides.svg" alt="How reckoner decides: pass 1 merges on a shared ID; pass 2 links exact normalized names at 0.60 unless the records carry conflicting IDs, in which case it refuses with a reason" width="880"></p>
|
|
96
|
+
|
|
97
|
+
## Why it works this way
|
|
98
|
+
|
|
99
|
+
The same company is spelled differently in every system it appears in:
|
|
100
|
+
|
|
101
|
+
| One source says | Another says |
|
|
102
|
+
| --- | --- |
|
|
103
|
+
| `GENERAL ELECTRIC CO` | `GENERAL ELECTRIC COMPANY` |
|
|
104
|
+
| `BECTON DICKINSON & CO` | `BECTON, DICKINSON AND COMPANY` |
|
|
105
|
+
| `THE SHERWIN-WILLIAMS COMPANY` | `Sherwin Williams Co` |
|
|
106
|
+
| `Medtronic plc` | `MEDTRONIC INC` |
|
|
107
|
+
|
|
108
|
+
Join two datasets on the raw name and you silently drop many true matches. Turn on a fuzzy matcher instead and you start quietly merging companies that are not the same, with no way to explain any single decision. reckoner takes the opposite stance:
|
|
109
|
+
|
|
110
|
+
- **Identifiers merge first.** A shared CIK, UEI, EIN, CAGE, ticker or domain is the only thing that merges two records outright. Registry IDs (CIK, UEI, EIN, CAGE) link at 0.97 to 0.99; reassignable ones (ticker 0.90, domain 0.85) lower.
|
|
111
|
+
- **Names link only on an exact match after conservative cleanup.** Case, punctuation, `&`, a leading `The`, and trailing legal suffixes (`Inc`, `Corp`, `Company`, `LLC`, `GmbH`, `plc`, ...) are removed. Distinguishing words are never dropped. Such links score 0.60.
|
|
112
|
+
- **No fuzzy scoring anywhere.** A typo does not match. There is no similarity knob to tune.
|
|
113
|
+
- **Conflicting registry IDs block a merge.** Same normalized name but different CIKs: refused, with the reason recorded.
|
|
114
|
+
- **Everything is a receipt.** Every link and refusal is a `MatchRecord` with the original values, the normalized value, the method, the confidence and the evidence.
|
|
115
|
+
|
|
116
|
+
It does not do fuzzy, edit-distance or embedding matching, machine learning, or network calls. If you need probabilistic linkage across millions of noisy consumer records, use a heavier library. reckoner is for when a wrong merge is worse than a missed one and every decision has to be explainable to an analyst, an auditor or a regulator.
|
|
117
|
+
|
|
118
|
+
Pure Python standard library, zero dependencies, one file of logic ([`reckoner/resolver.py`](reckoner/resolver.py)) you can read in ten minutes.
|
|
119
|
+
|
|
120
|
+
<details>
|
|
121
|
+
<summary><b>Command line reference</b></summary>
|
|
122
|
+
|
|
123
|
+
```
|
|
124
|
+
reckoner [-h] [-f {summary,json,csv}] [-o OUTPUT] [--agency] [--demo]
|
|
125
|
+
[--color {auto,always,never}] [--version] [input]
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
| Option | What it does |
|
|
129
|
+
| --- | --- |
|
|
130
|
+
| `input` | CSV, JSON array or JSON Lines file, or `-` for stdin |
|
|
131
|
+
| `-f summary` | readable report (default) |
|
|
132
|
+
| `-f json` | the full result with every match receipt |
|
|
133
|
+
| `-f csv` | each input row plus `entity_id`, `canonical_name`, `link_confidence` |
|
|
134
|
+
| `-o FILE` | write to a file instead of the terminal |
|
|
135
|
+
| `--agency` | government agency names: strips `U.S.` prefixes and expands `DoD`, `EPA`, ... |
|
|
136
|
+
| `--color` | `auto` (default: color in a terminal, off when piped or when `NO_COLOR` is set), `always`, `never` |
|
|
137
|
+
|
|
138
|
+
Recognised columns (case-insensitive): `name`, `agency`, `cik`, `uei`, `ein`, `cage`, `ticker`, `domain`, `source`, `state`. Other columns are kept in CSV output. If none of your columns are recognised, reckoner stops and tells you which columns it found and what to rename.
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
reckoner records.jsonl --format json # full result with every receipt
|
|
142
|
+
reckoner agencies.csv --agency # DoD, EPA, U.S. prefixes
|
|
143
|
+
cat companies.csv | reckoner - # read stdin
|
|
144
|
+
```
|
|
145
|
+
</details>
|
|
146
|
+
|
|
147
|
+
<details>
|
|
148
|
+
<summary><b>Use it from Python</b></summary>
|
|
149
|
+
|
|
150
|
+
```python
|
|
151
|
+
from reckoner import EntityResolver
|
|
152
|
+
|
|
153
|
+
records = [
|
|
154
|
+
{"name": "GENERAL ELECTRIC CO", "cik": "0000040545", "source": "sec"},
|
|
155
|
+
{"name": "General Electric Company", "source": "usaspending"},
|
|
156
|
+
{"name": "THE SHERWIN-WILLIAMS COMPANY", "source": "usaspending"},
|
|
157
|
+
{"name": "Sherwin Williams Co", "source": "sec"},
|
|
158
|
+
{"name": "Meta Platforms, Inc.", "cik": "1326801", "source": "sec"},
|
|
159
|
+
{"name": "Meta Financial Group, Inc.", "cik": "907471", "source": "sec"},
|
|
160
|
+
]
|
|
161
|
+
|
|
162
|
+
result = EntityResolver().resolve(records)
|
|
163
|
+
|
|
164
|
+
for e in result["entities"]:
|
|
165
|
+
print(e["canonical_name"], "<-", e["aliases"], f"(conf {e['link_confidence']})")
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
```
|
|
169
|
+
General Electric Company <- ['GENERAL ELECTRIC CO', 'General Electric Company'] (conf 0.6)
|
|
170
|
+
THE SHERWIN-WILLIAMS COMPANY <- ['THE SHERWIN-WILLIAMS COMPANY', 'Sherwin Williams Co'] (conf 0.6)
|
|
171
|
+
Meta Platforms, Inc. <- ['Meta Platforms, Inc.'] (conf 1.0)
|
|
172
|
+
Meta Financial Group, Inc. <- ['Meta Financial Group, Inc.'] (conf 1.0)
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
Both GE and Sherwin-Williams link on the normalized name (only one GE record carries the CIK, so the CIK cannot prove the link). The two Metas normalize to different names and carry different CIKs, so they never link.
|
|
176
|
+
|
|
177
|
+
When two records normalize to the same name but carry conflicting identifiers, the merge is refused and the refusal is kept:
|
|
178
|
+
|
|
179
|
+
```python
|
|
180
|
+
result = EntityResolver().resolve([
|
|
181
|
+
{"name": "Meta Platforms, Inc.", "cik": "1326801"},
|
|
182
|
+
{"name": "META PLATFORMS INC", "cik": "907471"},
|
|
183
|
+
])
|
|
184
|
+
result["entities_out"] # 2, with a refusal receipt in result["matches"]
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
Agency names: `EntityResolver(agency_mode=True)` adds US-prefix stripping and a small alias table (`DoD` to `department of defense`, `EPA` to `environmental protection agency`, ...):
|
|
188
|
+
|
|
189
|
+
```python
|
|
190
|
+
EntityResolver(agency_mode=True).resolve([
|
|
191
|
+
{"agency": "DoD"},
|
|
192
|
+
{"agency": "U.S. Department of Defense"},
|
|
193
|
+
]) # -> 1 entity
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
Run the full example with `PYTHONPATH=. python examples/quickstart.py` from the repo root.
|
|
197
|
+
</details>
|
|
198
|
+
|
|
199
|
+
<details>
|
|
200
|
+
<summary><b>The output shape</b></summary>
|
|
201
|
+
|
|
202
|
+
`resolve(records)` returns a dict:
|
|
203
|
+
|
|
204
|
+
```python
|
|
205
|
+
{
|
|
206
|
+
"entities": [ ... ], # one canonical entity per cluster
|
|
207
|
+
"clusters": [[0, 1], [2, 3], ...], # input indices grouped
|
|
208
|
+
"matches": [ ... ], # every MatchRecord, merged and refused
|
|
209
|
+
"records_in": 6,
|
|
210
|
+
"entities_out": 4,
|
|
211
|
+
}
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
Each entity:
|
|
215
|
+
|
|
216
|
+
```python
|
|
217
|
+
{
|
|
218
|
+
"entity_id": "ent-0",
|
|
219
|
+
"canonical_name": "General Electric Company",
|
|
220
|
+
"normalized_name": "general electric",
|
|
221
|
+
"identifiers": {"cik": "40545"},
|
|
222
|
+
"aliases": ["GENERAL ELECTRIC CO", "General Electric Company"],
|
|
223
|
+
"member_indices": [0, 1],
|
|
224
|
+
"sources": ["sec", "usaspending"],
|
|
225
|
+
"link_confidence": 0.6,
|
|
226
|
+
}
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
`link_confidence` is the strongest evidence that actually merged the cluster: an identifier's confidence if an identifier shared between members merged them, otherwise 0.60 for a name link. A single-record entity is 1.0.
|
|
230
|
+
|
|
231
|
+
Each match or refusal receipt:
|
|
232
|
+
|
|
233
|
+
```python
|
|
234
|
+
{
|
|
235
|
+
"left_index": 0, "right_index": 1,
|
|
236
|
+
"method": "name_exact_normalized",
|
|
237
|
+
"confidence": 0.0, "merged": False,
|
|
238
|
+
"left_original": "Meta Platforms, Inc.",
|
|
239
|
+
"right_original": "META PLATFORMS INC",
|
|
240
|
+
"normalized": "meta platforms",
|
|
241
|
+
"evidence": [],
|
|
242
|
+
"refusal_reason": "identical normalized names but conflicting CIK - refusing to merge",
|
|
243
|
+
}
|
|
244
|
+
```
|
|
245
|
+
</details>
|
|
246
|
+
|
|
247
|
+
<details>
|
|
248
|
+
<summary><b>Extending it, and running the tests</b></summary>
|
|
249
|
+
|
|
250
|
+
The normalization is deliberately small and readable. Add legal suffixes to `LEGAL_SUFFIXES`, agency aliases to `AGENCY_ALIASES`, or new identifier types to `IDENTIFIERS` with a normalizer and a confidence. Everything is exact-match on the normalized value; there is nothing fuzzy to tune.
|
|
251
|
+
|
|
252
|
+
```bash
|
|
253
|
+
git clone https://github.com/Mattbusel/reckoner
|
|
254
|
+
cd reckoner
|
|
255
|
+
python -m unittest discover -s tests
|
|
256
|
+
python -m reckoner --demo
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
Or just copy `reckoner/resolver.py` into your project. It is standard library only.
|
|
260
|
+
</details>
|
|
261
|
+
|
|
262
|
+
## License
|
|
263
|
+
|
|
264
|
+
MIT, see [LICENSE](LICENSE).
|
|
265
|
+
|
|
266
|
+
---
|
|
267
|
+
|
|
268
|
+
Built by the team behind [Tensorust](https://tensorust-site.vercel.app/), where reconciling public records across systems that each spell reality slightly differently is the whole job.
|