bibauthors 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- bibauthors-0.1.0/LICENSE +21 -0
- bibauthors-0.1.0/PKG-INFO +542 -0
- bibauthors-0.1.0/README.md +510 -0
- bibauthors-0.1.0/pyproject.toml +114 -0
- bibauthors-0.1.0/setup.cfg +4 -0
- bibauthors-0.1.0/src/bibauthors/__init__.py +108 -0
- bibauthors-0.1.0/src/bibauthors/cache.py +199 -0
- bibauthors-0.1.0/src/bibauthors/cli.py +297 -0
- bibauthors-0.1.0/src/bibauthors/config.py +191 -0
- bibauthors-0.1.0/src/bibauthors/entry.py +142 -0
- bibauthors-0.1.0/src/bibauthors/http.py +380 -0
- bibauthors-0.1.0/src/bibauthors/latex.py +399 -0
- bibauthors-0.1.0/src/bibauthors/names.py +1424 -0
- bibauthors-0.1.0/src/bibauthors/parsers/__init__.py +148 -0
- bibauthors-0.1.0/src/bibauthors/parsers/bibtex.py +593 -0
- bibauthors-0.1.0/src/bibauthors/parsers/csljson.py +220 -0
- bibauthors-0.1.0/src/bibauthors/providers/__init__.py +239 -0
- bibauthors-0.1.0/src/bibauthors/providers/arxiv.py +209 -0
- bibauthors-0.1.0/src/bibauthors/providers/crossref.py +254 -0
- bibauthors-0.1.0/src/bibauthors/providers/openalex.py +212 -0
- bibauthors-0.1.0/src/bibauthors/providers/semanticscholar.py +160 -0
- bibauthors-0.1.0/src/bibauthors/report.py +371 -0
- bibauthors-0.1.0/src/bibauthors/verdict.py +153 -0
- bibauthors-0.1.0/src/bibauthors/verify.py +397 -0
- bibauthors-0.1.0/src/bibauthors.egg-info/PKG-INFO +542 -0
- bibauthors-0.1.0/src/bibauthors.egg-info/SOURCES.txt +33 -0
- bibauthors-0.1.0/src/bibauthors.egg-info/dependency_links.txt +1 -0
- bibauthors-0.1.0/src/bibauthors.egg-info/entry_points.txt +2 -0
- bibauthors-0.1.0/src/bibauthors.egg-info/requires.txt +6 -0
- bibauthors-0.1.0/src/bibauthors.egg-info/top_level.txt +1 -0
- bibauthors-0.1.0/tests/test_accuracy.py +645 -0
- bibauthors-0.1.0/tests/test_cli.py +478 -0
- bibauthors-0.1.0/tests/test_names.py +752 -0
- bibauthors-0.1.0/tests/test_parsers.py +837 -0
- bibauthors-0.1.0/tests/test_providers.py +615 -0
bibauthors-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Manu Nicholas Jacob
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,542 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: bibauthors
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Verify bibliography AUTHOR LISTS against authoritative metadata, so a chimeric citation cannot pass as a clean one.
|
|
5
|
+
Author: Manu Nicholas Jacob
|
|
6
|
+
Maintainer: Manu Nicholas Jacob
|
|
7
|
+
License: MIT
|
|
8
|
+
Project-URL: Homepage, https://github.com/manunicholasjacob/bibauthors
|
|
9
|
+
Project-URL: Source, https://github.com/manunicholasjacob/bibauthors
|
|
10
|
+
Project-URL: Issues, https://github.com/manunicholasjacob/bibauthors/issues
|
|
11
|
+
Project-URL: Changelog, https://github.com/manunicholasjacob/bibauthors/blob/main/CHANGELOG.md
|
|
12
|
+
Project-URL: ORCID, https://orcid.org/0009-0007-6589-6572
|
|
13
|
+
Keywords: bibliography,bibtex,citations,author verification,hallucination,crossref,openalex,arxiv,reproducibility,latex
|
|
14
|
+
Classifier: Development Status :: 4 - Beta
|
|
15
|
+
Classifier: Intended Audience :: Science/Research
|
|
16
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
23
|
+
Classifier: Topic :: Text Processing :: Markup :: LaTeX
|
|
24
|
+
Classifier: Topic :: Scientific/Engineering
|
|
25
|
+
Requires-Python: >=3.9
|
|
26
|
+
Description-Content-Type: text/markdown
|
|
27
|
+
License-File: LICENSE
|
|
28
|
+
Requires-Dist: tomli>=1.2; python_version < "3.11"
|
|
29
|
+
Provides-Extra: dev
|
|
30
|
+
Requires-Dist: pytest; extra == "dev"
|
|
31
|
+
Dynamic: license-file
|
|
32
|
+
|
|
33
|
+
# bibauthors
|
|
34
|
+
|
|
35
|
+
Verify the **author list** of every entry in a bibliography against
|
|
36
|
+
authoritative metadata, and fail when the authors do not belong to the work.
|
|
37
|
+
|
|
38
|
+
Deliberately narrow. Every citation checker matches on title and DOI, and
|
|
39
|
+
most of them do that well. This one checks the author list, which is the part
|
|
40
|
+
they all do worst.
|
|
41
|
+
|
|
42
|
+
> A `DISJOINT` verdict is a prompt to check an entry by hand. It is **not** an
|
|
43
|
+
> accusation of fabrication. See [Limitations](#limitations).
|
|
44
|
+
|
|
45
|
+
- [The chimeric citation](#the-chimeric-citation)
|
|
46
|
+
- [Install](#install)
|
|
47
|
+
- [Usage](#usage)
|
|
48
|
+
- [Verdicts](#verdicts)
|
|
49
|
+
- [Thresholds](#thresholds)
|
|
50
|
+
- [Measured accuracy](#measured-accuracy)
|
|
51
|
+
- [Sources](#sources)
|
|
52
|
+
- [Related work](#related-work)
|
|
53
|
+
- [Limitations](#limitations)
|
|
54
|
+
|
|
55
|
+
## The chimeric citation
|
|
56
|
+
|
|
57
|
+
The most dangerous hallucinated citation is not the one that invents a paper.
|
|
58
|
+
An invented paper is easy to catch: no DOI resolves, no title matches, and
|
|
59
|
+
every existing tool flags it.
|
|
60
|
+
|
|
61
|
+
The dangerous one is the **chimeric citation**: a real title, a real venue, a
|
|
62
|
+
real year, and an **invented author list**. It passes a title-matching checker
|
|
63
|
+
with a clean report. It looks real on Google Scholar. A reader who recognises
|
|
64
|
+
the title has no reason to look twice, and a reader who recognises one of the
|
|
65
|
+
names has every reason to trust it.
|
|
66
|
+
|
|
67
|
+
This is not hypothetical. On 29 October 2026 one turned up in this author's
|
|
68
|
+
own bibliography. The entry for *Evaluating LLM Agent Collusion in Double
|
|
69
|
+
Auctions* (arXiv 2507.01413) carried the authors:
|
|
70
|
+
|
|
71
|
+
Agrawal, Liu, Fried, Sycara, Jacobs, Oesterheld, Conitzer
|
|
72
|
+
|
|
73
|
+
The real author list is:
|
|
74
|
+
|
|
75
|
+
Kushal Agrawal, Verona Teo, Juan J. Vazquez,
|
|
76
|
+
Sudarsh Kunnavakkam, Vishak Srikanth, Andy Liu
|
|
77
|
+
|
|
78
|
+
Two of the seven surnames are real authors of the paper. Five are not. The
|
|
79
|
+
title, the arXiv id, the venue and the year were all correct. The checker in
|
|
80
|
+
use at the time reported the entry as "suspicious, found in 1 source", which
|
|
81
|
+
is the same label it gives every legitimate arXiv preprint, so the label
|
|
82
|
+
carried no signal about the actual error.
|
|
83
|
+
|
|
84
|
+
`bibauthors` reports that entry like this:
|
|
85
|
+
|
|
86
|
+
```
|
|
87
|
+
FAIL collusion2025:9 PARTIAL_OVERLAP via arxiv
|
|
88
|
+
title: Evaluating LLM Agent Collusion in Double Auctions
|
|
89
|
+
record: http://arxiv.org/abs/2507.01413v1
|
|
90
|
+
cited (7):
|
|
91
|
+
[0] Agrawal, Kushal -> [0] Kushal Agrawal
|
|
92
|
+
[1] Liu, Andy -> [5] Andy Liu
|
|
93
|
+
[2] Fried, Daniel <-- NOT ON THIS WORK
|
|
94
|
+
[3] Sycara, Katia <-- NOT ON THIS WORK
|
|
95
|
+
[4] Jacobs, Abigail <-- NOT ON THIS WORK
|
|
96
|
+
[5] Oesterheld, Caspar <-- NOT ON THIS WORK
|
|
97
|
+
[6] Conitzer, Vincent <-- NOT ON THIS WORK
|
|
98
|
+
record authors not cited (4 of 6):
|
|
99
|
+
[1] Verona Teo
|
|
100
|
+
[2] Juan J. Vazquez
|
|
101
|
+
[3] Sudarsh Kunnavakkam
|
|
102
|
+
[4] Vishak Srikanth
|
|
103
|
+
coverage: 29% of cited authors matched
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
That entry is committed as `tests/fixtures/bib/chimeric.bib`, and the test
|
|
107
|
+
suite asserts the verdict on every run.
|
|
108
|
+
|
|
109
|
+
## Install
|
|
110
|
+
|
|
111
|
+
```sh
|
|
112
|
+
pip install bibauthors
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
`bibauthors` has **no required runtime dependencies**. It is standard library
|
|
116
|
+
only, so it drops into a minimal CI image. (`tomli` is pulled in only on
|
|
117
|
+
Python 3.9 and 3.10, and only if you use a TOML config file.) Python 3.9+.
|
|
118
|
+
|
|
119
|
+
### As a GitHub Action
|
|
120
|
+
|
|
121
|
+
This is the form most paper repositories want. Add one step:
|
|
122
|
+
|
|
123
|
+
```yaml
|
|
124
|
+
name: bibliography
|
|
125
|
+
on: [push, pull_request]
|
|
126
|
+
permissions:
|
|
127
|
+
contents: read
|
|
128
|
+
jobs:
|
|
129
|
+
authors:
|
|
130
|
+
runs-on: ubuntu-latest
|
|
131
|
+
steps:
|
|
132
|
+
- uses: actions/checkout@v4
|
|
133
|
+
- uses: manunicholasjacob/bibauthors@v0.1.0
|
|
134
|
+
with:
|
|
135
|
+
paths: refs.bib
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
Findings land as annotations on the pull request diff, at the line of the
|
|
139
|
+
offending entry. Inputs: `paths` (default `**/*.bib`), `offline`, `cache-dir`,
|
|
140
|
+
`providers`, `fail-on` (`mismatch`, `unverifiable` or `never`), `strict`,
|
|
141
|
+
`json-report`, `annotations`, `python-version`, `version`. Outputs:
|
|
142
|
+
`exit-code`, `mismatches`, `unverifiable`, `report-path`.
|
|
143
|
+
|
|
144
|
+
To keep CI hermetic and off the metadata APIs entirely, commit your cache and
|
|
145
|
+
run `offline: true` with `cache-dir` pointed at it.
|
|
146
|
+
|
|
147
|
+
### As a pre-commit hook
|
|
148
|
+
|
|
149
|
+
```yaml
|
|
150
|
+
repos:
|
|
151
|
+
- repo: https://github.com/manunicholasjacob/bibauthors
|
|
152
|
+
rev: v0.1.0
|
|
153
|
+
hooks:
|
|
154
|
+
- id: bibauthors
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
There is also `bibauthors-offline` for cache-only, network-free commits. Note
|
|
158
|
+
that pre-commit treats any non-zero exit as a failure, so an unverifiable
|
|
159
|
+
entry (exit 2) will block a commit there, unlike the Action's default policy.
|
|
160
|
+
|
|
161
|
+
## Usage
|
|
162
|
+
|
|
163
|
+
```sh
|
|
164
|
+
bibauthors refs.bib # human report
|
|
165
|
+
bibauthors refs.bib --json report.json # machine-readable too
|
|
166
|
+
bibauthors refs.bib --offline # cache only, no network
|
|
167
|
+
bibauthors refs.bib --strict # also fail on order and truncation
|
|
168
|
+
bibauthors refs.bib --set name_variant=0.75 # tune a threshold
|
|
169
|
+
bibauthors --list-thresholds # every knob and its default
|
|
170
|
+
cat refs.bib | bibauthors # reads stdin
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
Accepts BibTeX and CSL-JSON, sniffed by default.
|
|
174
|
+
|
|
175
|
+
### Exit codes
|
|
176
|
+
|
|
177
|
+
| Code | Meaning |
|
|
178
|
+
| ---- | ------- |
|
|
179
|
+
| 0 | clean: every entry `MATCH` or `MATCH_WITH_VARIANTS` |
|
|
180
|
+
| 1 | author mismatch found |
|
|
181
|
+
| 2 | no mismatch, but at least one entry could not be verified |
|
|
182
|
+
| 3 | usage or input error |
|
|
183
|
+
|
|
184
|
+
Code 1 wins over code 2 when both apply: the mismatch is the finding the tool
|
|
185
|
+
exists to report. Code 2 is separate on purpose. "I could not check this" and
|
|
186
|
+
"this is wrong" are different statements, and a checker that conflates them
|
|
187
|
+
teaches you to ignore it.
|
|
188
|
+
|
|
189
|
+
### As a library
|
|
190
|
+
|
|
191
|
+
```python
|
|
192
|
+
from bibauthors import Config, Verdict, verify_bibliography
|
|
193
|
+
|
|
194
|
+
report = verify_bibliography("refs.bib", Config(offline=True))
|
|
195
|
+
|
|
196
|
+
for result in report.results:
|
|
197
|
+
if result.verdict in (Verdict.DISJOINT, Verdict.PARTIAL_OVERLAP):
|
|
198
|
+
print(result.entry.key, result.verdict)
|
|
199
|
+
for index in result.comparison.unmatched_cited:
|
|
200
|
+
print(" not on this work:", result.entry.authors[index])
|
|
201
|
+
|
|
202
|
+
raise SystemExit(report.exit_code())
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
The name matcher is usable on its own, without any network or bibliography:
|
|
206
|
+
|
|
207
|
+
```python
|
|
208
|
+
from bibauthors import Thresholds, compare_names, parse_name
|
|
209
|
+
|
|
210
|
+
a = parse_name("van der Berg, J.")
|
|
211
|
+
b = parse_name("Jan van der Berg")
|
|
212
|
+
result = compare_names(a, b, Thresholds())
|
|
213
|
+
print(result.score, result.variants) # 0.9 ['initials']
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
Stable public API: `verify_bibliography`, `verify_entries`, `verify_entry`,
|
|
217
|
+
`Verifier`, `load_bibliography`, `parse_bibtex`, `parse_csl_json`, `Entry`,
|
|
218
|
+
`parse_name`, `parse_names`, `fold`, `compare_names`, `compare_author_lists`,
|
|
219
|
+
`PersonName`, `NameComparison`, `ListComparison`, `Verdict`, `Thresholds`,
|
|
220
|
+
`Config`, `EntryResult`, `Report`, `YearFinding`, `format_text`,
|
|
221
|
+
`format_json`, `format_github_annotations`, `to_dict`, `Provider`, `Record`,
|
|
222
|
+
`build_providers`, `HttpClient`, `Cache`.
|
|
223
|
+
|
|
224
|
+
The JSON report carries a `schema_version` (currently `1.0`), versioned
|
|
225
|
+
separately from the package. Within a major schema version, fields are added
|
|
226
|
+
but never renamed or removed.
|
|
227
|
+
|
|
228
|
+
## Verdicts
|
|
229
|
+
|
|
230
|
+
The verdict is graded, not boolean. A bibliography is full of legitimate
|
|
231
|
+
variation, and a tool that answers only yes or no either cries wolf or stays
|
|
232
|
+
silent.
|
|
233
|
+
|
|
234
|
+
| Verdict | Severity | Meaning | Fails? |
|
|
235
|
+
| ------- | -------- | ------- | ------ |
|
|
236
|
+
| `MATCH` | 0 | Cited authors and record authors agree exactly. | no |
|
|
237
|
+
| `MATCH_WITH_VARIANTS` | 0 | The same authors, written differently: initials, diacritics, particles, transliteration. | no |
|
|
238
|
+
| `TRUNCATED` | 1 | The cited list is a truncated prefix of the record, as `et al.` promises. | with `--strict` |
|
|
239
|
+
| `ORDER_DIFFERS` | 2 | The same authors, in a different order. Reported separately from a wrong author, because it usually means a transcription slip, not a fabrication. | with `--strict` |
|
|
240
|
+
| `UNVERIFIABLE` | 2 | No authoritative record was found to compare against. Not a finding about the entry. | exit 2 |
|
|
241
|
+
| `PARTIAL_OVERLAP` | 3 | Some cited authors are on the work and some are not. **This is where the real chimeric case lands**, because a fabricated list usually keeps a real name or two. | yes |
|
|
242
|
+
| `DISJOINT` | 4 | Not one cited author is on this work. | yes |
|
|
243
|
+
|
|
244
|
+
`DISJOINT` is deliberately hard to reach. It requires that nothing matched at
|
|
245
|
+
all, that the record lists at least `disjoint_min_record_authors` authors, that
|
|
246
|
+
the entry was not marked truncated, and that the record was identified by a
|
|
247
|
+
DOI or arXiv id or by a title similarity at or above `title_similarity`. If
|
|
248
|
+
nothing matched but any of those guards fails, the verdict falls back to
|
|
249
|
+
`PARTIAL_OVERLAP` with a note explaining the downgrade. A record found by
|
|
250
|
+
search with a weak title match is never allowed to drive a mismatch verdict
|
|
251
|
+
at all, because judging an author list against the wrong paper is the fastest
|
|
252
|
+
way to manufacture a false accusation.
|
|
253
|
+
|
|
254
|
+
Dates are a separate axis, reported as notes rather than verdicts. An arXiv
|
|
255
|
+
entry whose year matches the latest version but not v1 is surfaced like this:
|
|
256
|
+
|
|
257
|
+
```
|
|
258
|
+
year: stored year 2025 matches the latest version (2025-01-03),
|
|
259
|
+
but v1 is 2024 (2024-12-19). Decide which version you mean.
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
That is the second real case from the 29 October pass, committed as
|
|
263
|
+
`tests/fixtures/bib/wrong_year.bib` (the Qwen2.5 Technical Report, arXiv
|
|
264
|
+
2412.15115: v1 19 December 2024, v2 3 January 2025).
|
|
265
|
+
|
|
266
|
+
## Thresholds
|
|
267
|
+
|
|
268
|
+
Everything is configurable, from a `[tool.bibauthors]` table in
|
|
269
|
+
`pyproject.toml`, from a `bibauthors.toml`, or from repeatable `--set
|
|
270
|
+
NAME=VALUE` flags. `bibauthors --list-thresholds` prints them all.
|
|
271
|
+
|
|
272
|
+
| Threshold | Default | What it does |
|
|
273
|
+
| --------- | ------- | ------------ |
|
|
274
|
+
| `name_match` | 0.92 | At or above this, two names are the same with no variant worth reporting. |
|
|
275
|
+
| `name_variant` | 0.70 | At or above this, two names are the same person, reached by one or more variant rules. Below it, they are different people. |
|
|
276
|
+
| `partial_coverage` | 0.50 | Fraction of cited authors that must match before coverage is called adequate. |
|
|
277
|
+
| `title_similarity` | 0.80 | How close a record's title must be before the record may drive a mismatch verdict. |
|
|
278
|
+
| `disjoint_min_record_authors` | 2 | A record with fewer known authors can never produce `DISJOINT`. |
|
|
279
|
+
| `disjoint_max_coverage` | 0.0 | Coverage must be at or below this for `DISJOINT`. At the default, one matched author downgrades to `PARTIAL_OVERLAP`. |
|
|
280
|
+
| `surname_only_penalty` | 0.18 | Subtracted when one side has no given name, so a bare surname cannot reach `name_match` alone. |
|
|
281
|
+
| `initial_vs_full_given` | 0.90 | Ceiling for a given-name pair agreeing only on the initial. |
|
|
282
|
+
| `promote_order_differs` | false | Count `ORDER_DIFFERS` as a failure. |
|
|
283
|
+
| `promote_truncated` | false | Count `TRUNCATED` as a failure. |
|
|
284
|
+
|
|
285
|
+
`name_variant` is the consequential one. Lowering it hides real fabrications;
|
|
286
|
+
raising it manufactures them. The [measured accuracy](#measured-accuracy)
|
|
287
|
+
section includes a real sweep of it so the choice can be made on numbers
|
|
288
|
+
rather than taste.
|
|
289
|
+
|
|
290
|
+
The scoring model, for anyone who needs to reason about a number:
|
|
291
|
+
`score = 0.65 * family + 0.35 * given`, less `surname_only_penalty` when
|
|
292
|
+
either side has no given name, then clamped by the ceiling of every variant
|
|
293
|
+
rule that fired. The weights are load-bearing: a pair whose first given names
|
|
294
|
+
disagree can reach at most 0.65, which is below the default `name_variant`, so
|
|
295
|
+
"John Smith" against "Robert Smith" is a non-match on the given name alone no
|
|
296
|
+
matter how perfectly the surnames agree.
|
|
297
|
+
|
|
298
|
+
Variant rules and their ceilings: spacing and diacritics 1.0, dropped middle
|
|
299
|
+
name and dropped suffix 0.95, fuzzy group match 0.95, order swapped and
|
|
300
|
+
particle moved and German `ue` expansion and general spelling 0.90, initials
|
|
301
|
+
`initial_vs_full_given`, Mc/Mac 0.88, compound surname 0.85, conflicting
|
|
302
|
+
suffix and conflicting middle initial 0.85, transliteration 0.80, vowel
|
|
303
|
+
romanization 0.78.
|
|
304
|
+
|
|
305
|
+
## Measured accuracy
|
|
306
|
+
|
|
307
|
+
**These numbers come from a test set this project built. They are not an
|
|
308
|
+
independent benchmark, and no figure here is borrowed from any other
|
|
309
|
+
project.** Reproduce them with:
|
|
310
|
+
|
|
311
|
+
```sh
|
|
312
|
+
PYTHONPATH=src python tools/evaluate.py
|
|
313
|
+
```
|
|
314
|
+
|
|
315
|
+
### The set
|
|
316
|
+
|
|
317
|
+
| | |
|
|
318
|
+
| --- | --- |
|
|
319
|
+
| Name pairs | **154** labelled pairs: 89 the same human being, 65 different people |
|
|
320
|
+
| Categories | 19: exact, initials, middle_name, middle_initial_conflict, particle, mc_mac, apostrophe, hyphenated, diacritics, transliteration, cjk_order, cjk_script, suffix, comma_order, group, near_miss_surname, different_person, same_surname_different_person, initial_collision |
|
|
321
|
+
| Entries | **27** labelled bibliography entries, covering all 7 verdicts (5 MATCH, 5 MATCH_WITH_VARIANTS, 3 ORDER_DIFFERS, 4 TRUNCATED, 3 PARTIAL_OVERLAP, 5 DISJOINT, 2 UNVERIFIABLE) |
|
|
322
|
+
| Ground truth | Author lists from recorded CrossRef, OpenAlex, arXiv and Semantic Scholar responses, plus public record for the confusable-people pairs |
|
|
323
|
+
| Provenance | `tests/fixtures/eval/README-DATA.txt` |
|
|
324
|
+
|
|
325
|
+
The set is **small and deliberately weighted towards hard cases**. 34 of the
|
|
326
|
+
65 negative pairs are real confusable people or genuinely distinct near-miss
|
|
327
|
+
surnames; there are no easy negatives. One row moves pair accuracy by 0.65 of
|
|
328
|
+
a percentage point. **These figures are not an estimate of accuracy on an
|
|
329
|
+
ordinary bibliography, where the tool scores considerably higher.** They are a
|
|
330
|
+
measurement of where it breaks.
|
|
331
|
+
|
|
332
|
+
### Results, at the shipped defaults
|
|
333
|
+
|
|
334
|
+
| Metric | Name pairs (n=154) | Entries (n=27) |
|
|
335
|
+
| ------ | ------------------ | -------------- |
|
|
336
|
+
| Accuracy | 0.8766 (135/154) | 0.9630 (26/27) |
|
|
337
|
+
| Precision | 0.8571 | |
|
|
338
|
+
| Recall | 0.9438 | |
|
|
339
|
+
| F1 | 0.8984 | |
|
|
340
|
+
| **False-accusation rate** | **0.0562** (5 of 89) | **0.0588** (1 of 17) |
|
|
341
|
+
| **Miss rate** | **0.2154** (14 of 65) | **0.0000** (0 of 8) |
|
|
342
|
+
|
|
343
|
+
The last two rows are the ones that matter, and they are reported separately
|
|
344
|
+
and never averaged into one figure, because the costs are not symmetric. The
|
|
345
|
+
false-accusation rate is how often something genuinely correct is called
|
|
346
|
+
wrong. The miss rate is how often something genuinely wrong is not caught.
|
|
347
|
+
|
|
348
|
+
All 8 entries with a genuinely wrong author list were caught, including the
|
|
349
|
+
real chimeric case and four fully fabricated lists. One entry out of 17 with a
|
|
350
|
+
correct author list was wrongly given `PARTIAL_OVERLAP`, and the cause is
|
|
351
|
+
instructive: CrossRef deposits Aidan N. Gomez of *Attention Is All You Need*
|
|
352
|
+
with the family name `N.Gomez`, so a correctly spelled citation scores 0.909
|
|
353
|
+
against the record, just under the spelling floor. The defect is in the
|
|
354
|
+
deposited metadata, the bibliography is innocent, and the tool still reports
|
|
355
|
+
it. That is the shape most false accusations will take.
|
|
356
|
+
|
|
357
|
+
### Per category
|
|
358
|
+
|
|
359
|
+
| Category | n | Accuracy | | Category | n | Accuracy |
|
|
360
|
+
| -------- | - | -------- |-| -------- | - | -------- |
|
|
361
|
+
| apostrophe | 6 | 1.0000 | | initials | 10 | 1.0000 |
|
|
362
|
+
| cjk_order | 8 | 1.0000 | | mc_mac | 6 | 0.5000 |
|
|
363
|
+
| cjk_script | 6 | 0.8333 | | middle_initial_conflict | 6 | 0.3333 |
|
|
364
|
+
| comma_order | 6 | 1.0000 | | middle_name | 8 | 1.0000 |
|
|
365
|
+
| diacritics | 8 | 1.0000 | | near_miss_surname | 12 | 0.9167 |
|
|
366
|
+
| different_person | 12 | 1.0000 | | particle | 10 | 0.9000 |
|
|
367
|
+
| exact | 6 | 1.0000 | | same_surname_different_person | 10 | 1.0000 |
|
|
368
|
+
| group | 8 | 0.8750 | | suffix | 6 | 0.5000 |
|
|
369
|
+
| hyphenated | 8 | 0.7500 | | transliteration | 8 | 0.6250 |
|
|
370
|
+
| initial_collision | 10 | 1.0000 | | **ALL** | **154** | **0.8766** |
|
|
371
|
+
|
|
372
|
+
The three weak categories are honest and two of them are deliberate.
|
|
373
|
+
`middle_initial_conflict` (0.3333) and `suffix` (0.5000) are low because a
|
|
374
|
+
contradicting middle initial or generational suffix is scored as weak evidence
|
|
375
|
+
that two names are different people, never as proof. The consequence, measured:
|
|
376
|
+
George W. Bush and George H. W. Bush are treated as one person, as are the two
|
|
377
|
+
Michael Jordans, and so are John Smith Jr. and John Smith Sr. That is the cost
|
|
378
|
+
asymmetry doing exactly what it was set up to do, and it is a cost. `mc_mac`
|
|
379
|
+
(0.5000) is the same kind of choice applied to Mc/Mac. If you would rather pay
|
|
380
|
+
the other way, raise `name_variant` to 0.90; the sweep below shows what that
|
|
381
|
+
buys and what it costs.
|
|
382
|
+
|
|
383
|
+
### `name_variant` sweep, everything else held at default
|
|
384
|
+
|
|
385
|
+
| `name_variant` | Precision | Recall | F1 | False accusation | Miss |
|
|
386
|
+
| -------------- | --------- | ------ | -- | ---------------- | ---- |
|
|
387
|
+
| 0.50 | 0.6591 | 0.9775 | 0.7873 | 0.0225 | 0.6923 |
|
|
388
|
+
| 0.60 | 0.6591 | 0.9775 | 0.7873 | 0.0225 | 0.6923 |
|
|
389
|
+
| 0.65 | 0.7131 | 0.9775 | 0.8246 | 0.0225 | 0.5385 |
|
|
390
|
+
| **0.70 (default)** | **0.8571** | **0.9438** | **0.8984** | **0.0562** | **0.2154** |
|
|
391
|
+
| 0.75 | 0.8571 | 0.9438 | 0.8984 | 0.0562 | 0.2154 |
|
|
392
|
+
| 0.80 | 0.8632 | 0.9213 | 0.8913 | 0.0787 | 0.2000 |
|
|
393
|
+
| 0.85 | 0.8681 | 0.8876 | 0.8778 | 0.1124 | 0.1846 |
|
|
394
|
+
| 0.90 | 1.0000 | 0.8764 | 0.9341 | 0.1236 | 0.0000 |
|
|
395
|
+
| 0.95 | 1.0000 | 0.5843 | 0.7376 | 0.4157 | 0.0000 |
|
|
396
|
+
|
|
397
|
+
Worth reading honestly: on this set the default of 0.70 sits on a plateau
|
|
398
|
+
rather than at an optimum, and 0.90 reaches a zero miss rate at the cost of
|
|
399
|
+
more than doubling the false-accusation rate. The default prefers the cheaper
|
|
400
|
+
error. That is a result about 154 labelled pairs, not a law.
|
|
401
|
+
|
|
402
|
+
### Test suite
|
|
403
|
+
|
|
404
|
+
```
|
|
405
|
+
555 passed, 21 xfailed in 5.44s
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
The whole suite runs **offline** against recorded API responses in
|
|
409
|
+
`tests/fixtures/cache`, so it is reproducible and never touches a live API.
|
|
410
|
+
The 21 `xfailed` are the known failures above, each marked
|
|
411
|
+
`xfail(strict=True)` with its reason, so any future fix flips it to `XPASS`
|
|
412
|
+
and is impossible to miss. Failures are recorded, not hidden.
|
|
413
|
+
|
|
414
|
+
## Sources
|
|
415
|
+
|
|
416
|
+
| Source | Used for | Notes |
|
|
417
|
+
| ------ | -------- | ----- |
|
|
418
|
+
| **CrossRef** | DOI lookup, and `query.bibliographic` without one | The most reliable path for published work. |
|
|
419
|
+
| **arXiv** | Preprints | The right source for the v1 versus later-version date problem. HTTPS only. |
|
|
420
|
+
| **OpenAlex** | Broadest coverage | Works with no API key (verified 1 Nov 2026). |
|
|
421
|
+
| **Semantic Scholar** | Secondary corroboration | Rate-limits hard; serialized, 3 s minimum interval, backoff on 429. |
|
|
422
|
+
| **DBLP** | not implemented | Behind a bot wall (Anubis); it does not answer programmatic fetches, so depending on it would be a liability. |
|
|
423
|
+
|
|
424
|
+
Operational facts learned the hard way and encoded in the code:
|
|
425
|
+
|
|
426
|
+
- arXiv DOIs under the `10.48550/` prefix are generally **not in CrossRef**,
|
|
427
|
+
because they are registered with DataCite. A live
|
|
428
|
+
`GET /works/10.48550/arXiv.2507.01413` returns HTTP 404. `bibauthors` treats
|
|
429
|
+
that as "wrong registry, no information", never as a failed verification.
|
|
430
|
+
- Metadata sources disagree with each other. For the same Qwen2.5 record,
|
|
431
|
+
arXiv emits the deposited group prefix `Qwen` and a bare `:` as two separate
|
|
432
|
+
author elements, while OpenAlex reverses one name. Semantic Scholar
|
|
433
|
+
abbreviates: it returns `J. Vazquez` where arXiv returns `Juan J. Vazquez`.
|
|
434
|
+
None of this is normalized away, because hiding it would hide real
|
|
435
|
+
disagreement from the matcher.
|
|
436
|
+
- A CrossRef bibliographic search for *Attention Is All You Need* returns five
|
|
437
|
+
records with identical titles, the correct author list, and **2025** issue
|
|
438
|
+
dates under a reprint DOI prefix, not the 2017 original. This is why a
|
|
439
|
+
search hit is never treated as an identity.
|
|
440
|
+
|
|
441
|
+
`bibauthors` is a polite client: a real User-Agent carrying a **contact URL**,
|
|
442
|
+
per-host rate limiting, an on-disk cache, and full offline operation.
|
|
443
|
+
|
|
444
|
+
### A note on data exfiltration
|
|
445
|
+
|
|
446
|
+
OpenAlex responses carry an injected `meta.notice` field whose text instructs
|
|
447
|
+
an automated client to obtain an API key by POSTing **its user's email
|
|
448
|
+
address** to an endpoint. `bibauthors` treats that text as untrusted data,
|
|
449
|
+
never acts on it, and strips it before the response reaches the cache or a
|
|
450
|
+
log. There is no code path in this project that can put an email address into
|
|
451
|
+
an outbound request, and a test enforces that property. If you add a provider,
|
|
452
|
+
keep it.
|
|
453
|
+
|
|
454
|
+
## Related work
|
|
455
|
+
|
|
456
|
+
All five projects below were checked against the GitHub API on **1 November
|
|
457
|
+
2026**; star counts and dates age. **The source code of these projects was
|
|
458
|
+
deliberately not read, copied, adapted or paraphrased.** `bibauthors` was
|
|
459
|
+
implemented from the metadata providers' public API documentation and its own
|
|
460
|
+
design. What follows describes each project from its own public description
|
|
461
|
+
only, and makes no claim about how any of them works internally.
|
|
462
|
+
|
|
463
|
+
| Project | Language | Stars | License | Described by its authors as |
|
|
464
|
+
| ------- | -------- | ----- | ------- | -------------------------- |
|
|
465
|
+
| [gianlucasb/hallucinator](https://github.com/gianlucasb/hallucinator) | Rust | 391 | not SPDX-identified | detecting potentially hallucinated or fabricated references in academic PDF papers |
|
|
466
|
+
| [merfanian/Bibtex-Verifier](https://github.com/merfanian/Bibtex-Verifier) | JavaScript | 138 | MIT | a local web app verifying `.bib` entries against academic databases |
|
|
467
|
+
| [davidjurgens/hallucinated-reference-finder](https://github.com/davidjurgens/hallucinated-reference-finder) | Python | 44 | none declared | no description published |
|
|
468
|
+
| [PHY041/claude-skill-citation-checker](https://github.com/PHY041/claude-skill-citation-checker) | Python | 40 | none declared | a citation hallucination detector verifying `.bib` files against CrossRef, Semantic Scholar and OpenAlex |
|
|
469
|
+
| [seun-john/verify-references](https://github.com/seun-john/verify-references) | Python | 0 | MIT | detecting hallucinated and mis-cited references against Crossref, OpenAlex, PubMed, CORE and DataCite |
|
|
470
|
+
|
|
471
|
+
There are also several commercial web services that check reference lists for
|
|
472
|
+
fabrication. They are closed source, so there is nothing to compare against
|
|
473
|
+
beyond their marketing.
|
|
474
|
+
|
|
475
|
+
**Where `bibauthors` differs.** The shared design of this space is
|
|
476
|
+
title-and-identifier matching with a per-entry confidence label. That design
|
|
477
|
+
catches an invented paper reliably and is largely blind to a chimeric one,
|
|
478
|
+
because the title and the identifier are correct. The concrete observation
|
|
479
|
+
that motivated this project is in [the opening section](#the-chimeric-citation):
|
|
480
|
+
a real chimeric entry, whose author list was five-sevenths invented, was
|
|
481
|
+
reported as "suspicious, found in 1 source" by the checker in use, which is
|
|
482
|
+
the same label that tool gives every legitimate arXiv preprint. The label was
|
|
483
|
+
not wrong, it just was not about the error.
|
|
484
|
+
|
|
485
|
+
So `bibauthors` does one thing those tools do not foreground: it treats the
|
|
486
|
+
author list as the object under test, grades the comparison instead of
|
|
487
|
+
labelling it, reports author-order permutation as a finding distinct from a
|
|
488
|
+
wrong author, and publishes its own measured false-accusation and miss rates
|
|
489
|
+
so the tradeoff is inspectable. It is a complement to a title-and-DOI checker,
|
|
490
|
+
not a replacement for one. Run both.
|
|
491
|
+
|
|
492
|
+
## Limitations
|
|
493
|
+
|
|
494
|
+
**A `DISJOINT` verdict is a prompt to check an entry by hand. It is not an
|
|
495
|
+
accusation of fabrication, and nothing in this tool's output should be quoted
|
|
496
|
+
as evidence that anyone fabricated anything.** Metadata is imperfect, author
|
|
497
|
+
lists change between preprint and proceedings, records get deposited with
|
|
498
|
+
mangled names, and a search hit can simply be the wrong work. The measured
|
|
499
|
+
false-accusation rate above is not zero and will never be zero. A human
|
|
500
|
+
decides what a finding means.
|
|
501
|
+
|
|
502
|
+
Known limitations, all of them measured rather than guessed:
|
|
503
|
+
|
|
504
|
+
- **A contradicting middle initial or generational suffix does not separate
|
|
505
|
+
two people.** Measured: the two Presidents Bush, the two Michael Jordans,
|
|
506
|
+
and a Jr./Sr. pair each read as one person. This is a deliberate choice
|
|
507
|
+
under the cost asymmetry, and it is a real cost. Raise `name_variant` to
|
|
508
|
+
0.90 to pay the other way.
|
|
509
|
+
- **Mc and Mac are treated as one surname** at a 0.88 ceiling. Plenty of
|
|
510
|
+
McDonalds and MacDonalds are unrelated.
|
|
511
|
+
- **Scripts are not bridged.** A name in Han characters scores 0.0 against its
|
|
512
|
+
own romanization: the tool refuses to guess rather than risk a wrong match.
|
|
513
|
+
CJK name *ordering* is handled, in both romanized and Han forms, but a
|
|
514
|
+
Chinese-language bibliography entry for a real author is unmatchable.
|
|
515
|
+
- **Given-name equivalence across languages is not handled.** Lev and Leo
|
|
516
|
+
Tolstoy read as different people.
|
|
517
|
+
- **Transliteration handling is narrow by design** and documented as
|
|
518
|
+
incomplete in the source. It catches digraph substitutions and single-vowel
|
|
519
|
+
romanization differences. It cannot distinguish `Johnson`/`Johnsen` (two
|
|
520
|
+
surnames) from `Gorbachev`/`Gorbachov` (one), because the two pairs are
|
|
521
|
+
structurally identical, and it currently merges both.
|
|
522
|
+
- **A dropped particle is not handled.** "Ludwig van Beethoven" against
|
|
523
|
+
"Ludwig Beethoven" reads as different people. A particle *moved* behind the
|
|
524
|
+
given name is handled.
|
|
525
|
+
- **`compare_names` scores are not perfectly symmetric**, because
|
|
526
|
+
`difflib.SequenceMatcher.ratio` is order-dependent. No decision in the test
|
|
527
|
+
set changes, and a test pins the exact set of affected pairs so a new
|
|
528
|
+
asymmetry cannot hide.
|
|
529
|
+
- **An entry with no author field is not a finding.** A book cited by editor
|
|
530
|
+
parses with an empty author list and a warning, not a mismatch.
|
|
531
|
+
- **Coverage is only as good as the sources.** Books, theses, standards,
|
|
532
|
+
technical reports and anything not in CrossRef, OpenAlex, arXiv or Semantic
|
|
533
|
+
Scholar come back `UNVERIFIABLE`, which is a statement about the sources and
|
|
534
|
+
not about the citation.
|
|
535
|
+
|
|
536
|
+
## License and citation
|
|
537
|
+
|
|
538
|
+
MIT. Copyright (c) 2026 Manu Nicholas Jacob.
|
|
539
|
+
|
|
540
|
+
Author: **Manu Nicholas Jacob**, Independent Researcher
|
|
541
|
+
([ORCID 0009-0007-6589-6572](https://orcid.org/0009-0007-6589-6572)).
|
|
542
|
+
See `CITATION.cff`. Contact is via this repository's issue tracker.
|