uavocab 0.5.0__tar.gz

uavocab-0.5.0/LICENSES.md

@@ -0,0 +1,50 @@
+# Licenses
+
+## The combined work: CC-BY-NC-SA 4.0
+
+This package as a whole is distributed under the Creative Commons
+Attribution-NonCommercial-ShareAlike 4.0 International license
+(https://creativecommons.org/licenses/by-nc-sa/4.0/).
+
+Why NonCommercial and share-alike: the per-lemma CEFR levels come from the
+Ukrainian Vocabulary Profile (PULS), which its authors license CC BY-NC-SA 4.0
+(decided 2026-09-24) and which is redistributed here with their permission.
+Those levels are in the validated overlay, in every row of the crosswalk, in
+the `puls_level` column of the two Borodin tables, and in the `related_level`
+column of the aspect table. NonCommercial and share-alike propagate to a
+combined work, so the package carries both. They are the one NonCommercial
+component; nothing else NonCommercial is shipped. The frequency list and the
+two Borodin minimums are CC-BY 4.0 on their own and remain so, and the
+government and collocation tables and the stress column are CC-BY-SA 4.0; each
+dataset's own license is recorded in `uavocab.sources()` and below, so a downstream user who drops
+the PULS level columns owes attribution (and, for the corpus tables,
+share-alike) and nothing more.
+
+Cite the profile as its authors ask: Профіль української лексики (ПУЛЬС) / О. Синчак, В. Старко, М. Бурак, М. Свистун та ін. Львів: Школа української мови та культури УКУ, 2026.
+
+## Per dataset
+
+| Dataset | License | Attribution |
+|---|---|---|
+| `frequency` (top-12,000 lemma list) | CC-BY 4.0 | Derived aggregate counts from the UberText 2.0 lemma frequency dictionary, lang-uk. Chaplynskyi, D. (2023). Introducing UberText 2.0: A Corpus of Modern Ukrainian at Scale. Proc. 2nd UNLP, ACL. https://lang.org.ua/en/ubertext/ |
+| `validated_cefr` (PULS CEFR overlay) | CC-BY-NC-SA 4.0 | Ukrainian Vocabulary Profile (PULS): Synchak, O., Starko, V., Burak, M., Svystun, M. https://puls.peremova.org/ (eLex 2025). Per-lemma levels redistributed with the authors' permission (V. Starko, July 2026); CC BY-NC-SA 4.0 at the authors' decision (O. Synchak, 2026-09-24), attribution required. Snapshot 2026-07-23; the profile is not final. |
+| `borodin_a1` (A1 lexical minimum) | CC-BY 4.0 | Borodin, K. and Turkevych, O. 1000 i 1 slovo. Leksychnyi minimum z ukrainskoi movy yak inozemnoi. Riven A1. Slavic Language Education 3, LANGUAGE-LAB, Humboldt-Universitat zu Berlin. https://doi.org/10.18452/28236. Machine-readable list provided by the authors, 2026-07-27. The `puls_level` column is CC-BY-NC-SA (PULS). |
+| `borodin_a2` (A2 lexical minimum) | CC-BY 4.0 | Borodin, K. and Lazarenko, O. (2026). 1000 i 1 slovo. Riven A2. Europa-Universitat Viadrina Frankfurt (Oder). https://doi.org/10.11584/opus4-1448. The `puls_level` column is CC-BY-NC-SA (PULS). |
+| `crosswalk` (minimums against the profile) | CC-BY-NC-SA 4.0 | Lawrence, J. (2026), over the two Borodin volumes (CC-BY 4.0), the PULS profile (CC BY-NC-SA 4.0, by permission) and UberText 2.0 ranks (CC-BY 4.0). |
+| `b1_aspect` (aspect relations) | CC-BY-NC-SA 4.0 | Levels: PULS, by permission, CC BY-NC-SA 4.0. Aspect partners: Wiktionary via kaikki.org, CC-BY-SA 3.0. Counts: UberText 2.0, CC-BY 4.0. |
+| `b1_government` (case government) | CC-BY-SA 4.0 | Counts over the UberText 2.0 Wikipedia subcorpus (lang-uk, CC-BY 4.0); derived statistics CC-BY-SA 4.0. |
+| `b1_collocations` (logDice collocations) | CC-BY-SA 4.0 | Same corpus and terms as `b1_government`. |
+| `stress` (word stress per lemma) | CC-BY-SA 4.0 | ukrainian-word-stress 2.1.0 (lang-uk: Oleksiy Syvokon, maintained by Dmytro Chaplynskyi), MIT, https://github.com/lang-uk/ukrainian-word-stress; en.wiktionary via kaikki.org and uk.wiktionary, CC-BY-SA 3.0; uk.wikipedia lead sentences for a few names, CC-BY-SA 4.0; goroh.pp.ua consulted for 19 tiebreaks (attested stress only). Share-alike from the Wiktionary rows; its own table, joined on rank, so the list stays CC-BY. |
+
+## Not in this package
+
+Predicted CEFR levels (the site's gap-fill model; predictions are never merged
+into the validated overlay), any VESUM-derived lexicon or wordform data
+(CC BY-NC-SA), the UniMorph derivational families (CC BY-NC 4.0), the
+Wikipedia example sentences, and the Wiktionary paradigm tables. If you need
+any of them, they are on the site with their own terms.
+
+## The code
+
+The Python code in this package (`src/uavocab/*.py`) is released under the
+same CC-BY-NC-SA 4.0 terms as the data, to keep one license on the wheel.

uavocab-0.5.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: uavocab
+Version: 0.5.0
+Summary: The open Ukrainian vocabulary data behind uavocab.org, offline: the top-12,000 lemma frequency list, the validated PULS CEFR overlay, the Borodin A1 and A2 lexical minimums, their crosswalk, the B1 enrichment tables, and a per-lemma word-stress column.
+Author: Jake Lawrence
+License-Expression: CC-BY-NC-SA-4.0
+Project-URL: Homepage, https://uavocab.org/research
+Project-URL: Documentation, https://uavocab.org/research#release
+Project-URL: Data, https://uavocab.org/data
+Project-URL: Source, https://github.com/jake0lawrence/UA-Vocab
+Project-URL: Issues, https://github.com/jake0lawrence/UA-Vocab/issues
+Project-URL: Changelog, https://github.com/jake0lawrence/UA-Vocab/blob/main/python/uavocab/CHANGELOG.md
+Project-URL: Release files, https://github.com/jake0lawrence/UA-Vocab/tree/main/downloads/ukrainian-frequency
+Project-URL: Zenodo: frequency list, https://doi.org/10.5281/zenodo.22823134
+Project-URL: Zenodo: word stress, https://doi.org/10.5281/zenodo.23092388
+Keywords: ukrainian,uavocab,vocabulary,frequency list,CEFR,lexical minimum,language learning,corpus
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: Science/Research
+Classifier: Natural Language :: Ukrainian
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Education
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSES.md
+Requires-Dist: pyarrow>=14
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Dynamic: license-file
+
+# uavocab
+
+The open Ukrainian vocabulary data behind [uavocab.org](https://uavocab.org),
+installable and offline: the top-12,000 lemma frequency list over UberText 2.0,
+the validated PULS CEFR overlay (A1 to C1), the Borodin A1 and A2 lexical
+minimums, the crosswalk between the minimums and the profile, the B1
+enrichment tables (aspect relations, case government, collocations), and a
+per-lemma word-stress column. The study the data comes from, with its methods,
+findings and tools, is [uavocab.org/research](https://uavocab.org/research).
+
+```
+pip install uavocab
+```
+
+```python
+import uavocab
+uavocab.lookup("п'ять")["level"]                       # 'A1', with rank, source and enrichment beside it
+uavocab.lookup("орган")["stress"]                       # {'stress': 'о́рган', 'variants': ['о́рган', 'орга́н'], 'status': 'homograph', ...}
+uavocab.coverage(["в", "школа", "йти"], "A2")["coverage"]  # share of tokens a validated-A2 learner knows
+```
+
+This package was published as `ukvocab` up to 0.4.1. It was renamed to match
+the site it belongs to; `pip install ukvocab` still works and now installs
+this package, and `import ukvocab` keeps working with a warning that names
+the new import.
+
+## Where everything is
+
+| | |
+|---|---|
+| The study (methods, findings, tools) | https://uavocab.org/research |
+| Every dataset, its license and how to cite it | https://uavocab.org/data |
+| The release files (CSV, JSON, Anki, with SHA-256 sums) | https://uavocab.org/research#release |
+| Every release file, the datasheets, and this package's source (fork it) | https://github.com/jake0lawrence/UA-Vocab |
+| Archived copies with DOIs (Zenodo) | frequency list [10.5281/zenodo.22823134](https://doi.org/10.5281/zenodo.22823134), word stress [10.5281/zenodo.23092388](https://doi.org/10.5281/zenodo.23092388) |
+| The lemmatizer matched to this list | https://uavocab.org/research/lemmatizer |
+| PyPI | https://pypi.org/project/uavocab/ |
+
+## What you get
+
+| function | returns |
+|---|---|
+| `lookup(lemma, pos=None)` | one lemma: `level`, `level_source`, `rank`, the frequency figures, the `stress` (primary stressed form, accepted variants, resolution status and note), every other reading of the spelling, minimum membership, the crosswalk class, and the B1 `enrichment` (aspect partners, government, collocations); `None` when no table knows the string |
+| `levels(lemmas)` | batch: input string to `{level, level_source, rank, pos, source}` |
+| `coverage(lemmas, known_set)` | token coverage of a lemmatized text against a known set: a level (`"A2"` means every validated lemma at or below A2), `"borodin-a1"` / `"borodin-a2"`, an int N (top N of the list), or your own set of lemmas; reports the unknowns by frequency |
+| `sources()` | version, snapshot date, license, attribution, citation and SHA-256 per dataset, the combined license and why, and what is deliberately not shipped |
+
+Every return value carries the dataset, version and snapshot it came from.
+Data is Parquet, loaded lazily on first use, never on import. The only
+dependency is `pyarrow`.
+
+Two things the package does not do, on purpose. It does not lemmatize: pass
+lemmas in the UberText / VESUM convention (infinitive for verbs, nominative
+singular for nouns); the site's `POST /api/lemmatize` is the hosted lemmatizer
+matched to this list ([its contract](https://uavocab.org/research/lemmatizer)).
+And it ships no predicted CEFR levels: only the profile's
+validated levels are here, and a lemma the profile does not cover gets
+`level: None` rather than a guess.
+
+## Provenance and honesty
+
+The data tier is generated from the site's published release files by
+`scripts/build-package.py` in [the open repository](https://github.com/jake0lawrence/UA-Vocab),
+which holds every release file too, and is never edited by hand; the tests assert that every table's row count and hash
+match the published file exactly. `sources()` is the manifest that build
+wrote. The PULS overlay is a dated snapshot (2026-07-23) of a profile that is
+still being extended, redistributed with its authors' permission. The join
+onto the frequency list is part-of-speech aware; ten declared cross-tag rules
+recover the words the two tag conventions merely label differently, and each
+row's `join` field says how it matched, so the strict join is one filter away.
+
+## License
+
+The combined package is **CC-BY-NC-SA 4.0**, because the PULS levels are
+NonCommercial and share-alike at the authors' decision (2026-09-24) and they run
+through most tables. The frequency list and the two Borodin minimums are CC-BY
+4.0 on their own, and the government, collocation and stress tables CC-BY-SA
+4.0; drop the PULS level columns and NonCommercial does not apply. `LICENSES.md` has the
+per-dataset terms and every attribution.
+
+## Citation
+
+```
+Lawrence, J. (2026). uavocab: the Ukrainian Frequency datasets (top-12,000 lemma
+list, validated PULS CEFR overlay, Borodin A1/A2 minimums, crosswalk, B1
+enrichment tables, word stress), version 0.5.0. https://uavocab.org/research
+
+Upstream: Chaplynskyi, D. (2023). Introducing UberText 2.0: A Corpus of Modern
+Ukrainian at Scale. Proc. 2nd UNLP, ACL. | Synchak, O., Starko, V., Burak, M.,
+Svystun, M. (2025). Corpus-Based Vocabulary Profiling for Ukrainian: From
+Lexical Analysis to CEFR. eLex 2025; PULS, puls.peremova.org. | Borodin, K. and
+Turkevych, O. 1000 i 1 slovo, A1. https://doi.org/10.18452/28236 | Borodin, K.
+and Lazarenko, O. (2026). 1000 i 1 slovo, A2. https://doi.org/10.11584/opus4-1448
+| Syvokon, O. and Chaplynskyi, D. (lang-uk). ukrainian-word-stress, MIT.
+https://github.com/lang-uk/ukrainian-word-stress
+```
+
+Archived, DOI-minted copies of each dataset are listed in `sources()` under
+`conceptDoi` once they exist (today the frequency list and the stress column;
+the PULS-derived datasets wait on the profile's published license notice).
+Cite the concept DOI for the dataset you used.

uavocab-0.5.0/README.md

@@ -0,0 +1,101 @@
+# uavocab
+
+The open Ukrainian vocabulary data behind [uavocab.org](https://uavocab.org),
+installable and offline: the top-12,000 lemma frequency list over UberText 2.0,
+the validated PULS CEFR overlay (A1 to C1), the Borodin A1 and A2 lexical
+minimums, the crosswalk between the minimums and the profile, the B1
+enrichment tables (aspect relations, case government, collocations), and a
+per-lemma word-stress column. The study the data comes from, with its methods,
+findings and tools, is [uavocab.org/research](https://uavocab.org/research).
+
+```
+pip install uavocab
+```
+
+```python
+import uavocab
+uavocab.lookup("п'ять")["level"]                       # 'A1', with rank, source and enrichment beside it
+uavocab.lookup("орган")["stress"]                       # {'stress': 'о́рган', 'variants': ['о́рган', 'орга́н'], 'status': 'homograph', ...}
+uavocab.coverage(["в", "школа", "йти"], "A2")["coverage"]  # share of tokens a validated-A2 learner knows
+```
+
+This package was published as `ukvocab` up to 0.4.1. It was renamed to match
+the site it belongs to; `pip install ukvocab` still works and now installs
+this package, and `import ukvocab` keeps working with a warning that names
+the new import.
+
+## Where everything is
+
+| | |
+|---|---|
+| The study (methods, findings, tools) | https://uavocab.org/research |
+| Every dataset, its license and how to cite it | https://uavocab.org/data |
+| The release files (CSV, JSON, Anki, with SHA-256 sums) | https://uavocab.org/research#release |
+| Every release file, the datasheets, and this package's source (fork it) | https://github.com/jake0lawrence/UA-Vocab |
+| Archived copies with DOIs (Zenodo) | frequency list [10.5281/zenodo.22823134](https://doi.org/10.5281/zenodo.22823134), word stress [10.5281/zenodo.23092388](https://doi.org/10.5281/zenodo.23092388) |
+| The lemmatizer matched to this list | https://uavocab.org/research/lemmatizer |
+| PyPI | https://pypi.org/project/uavocab/ |
+
+## What you get
+
+| function | returns |
+|---|---|
+| `lookup(lemma, pos=None)` | one lemma: `level`, `level_source`, `rank`, the frequency figures, the `stress` (primary stressed form, accepted variants, resolution status and note), every other reading of the spelling, minimum membership, the crosswalk class, and the B1 `enrichment` (aspect partners, government, collocations); `None` when no table knows the string |
+| `levels(lemmas)` | batch: input string to `{level, level_source, rank, pos, source}` |
+| `coverage(lemmas, known_set)` | token coverage of a lemmatized text against a known set: a level (`"A2"` means every validated lemma at or below A2), `"borodin-a1"` / `"borodin-a2"`, an int N (top N of the list), or your own set of lemmas; reports the unknowns by frequency |
+| `sources()` | version, snapshot date, license, attribution, citation and SHA-256 per dataset, the combined license and why, and what is deliberately not shipped |
+
+Every return value carries the dataset, version and snapshot it came from.
+Data is Parquet, loaded lazily on first use, never on import. The only
+dependency is `pyarrow`.
+
+Two things the package does not do, on purpose. It does not lemmatize: pass
+lemmas in the UberText / VESUM convention (infinitive for verbs, nominative
+singular for nouns); the site's `POST /api/lemmatize` is the hosted lemmatizer
+matched to this list ([its contract](https://uavocab.org/research/lemmatizer)).
+And it ships no predicted CEFR levels: only the profile's
+validated levels are here, and a lemma the profile does not cover gets
+`level: None` rather than a guess.
+
+## Provenance and honesty
+
+The data tier is generated from the site's published release files by
+`scripts/build-package.py` in [the open repository](https://github.com/jake0lawrence/UA-Vocab),
+which holds every release file too, and is never edited by hand; the tests assert that every table's row count and hash
+match the published file exactly. `sources()` is the manifest that build
+wrote. The PULS overlay is a dated snapshot (2026-07-23) of a profile that is
+still being extended, redistributed with its authors' permission. The join
+onto the frequency list is part-of-speech aware; ten declared cross-tag rules
+recover the words the two tag conventions merely label differently, and each
+row's `join` field says how it matched, so the strict join is one filter away.
+
+## License
+
+The combined package is **CC-BY-NC-SA 4.0**, because the PULS levels are
+NonCommercial and share-alike at the authors' decision (2026-09-24) and they run
+through most tables. The frequency list and the two Borodin minimums are CC-BY
+4.0 on their own, and the government, collocation and stress tables CC-BY-SA
+4.0; drop the PULS level columns and NonCommercial does not apply. `LICENSES.md` has the
+per-dataset terms and every attribution.
+
+## Citation
+
+```
+Lawrence, J. (2026). uavocab: the Ukrainian Frequency datasets (top-12,000 lemma
+list, validated PULS CEFR overlay, Borodin A1/A2 minimums, crosswalk, B1
+enrichment tables, word stress), version 0.5.0. https://uavocab.org/research
+
+Upstream: Chaplynskyi, D. (2023). Introducing UberText 2.0: A Corpus of Modern
+Ukrainian at Scale. Proc. 2nd UNLP, ACL. | Synchak, O., Starko, V., Burak, M.,
+Svystun, M. (2025). Corpus-Based Vocabulary Profiling for Ukrainian: From
+Lexical Analysis to CEFR. eLex 2025; PULS, puls.peremova.org. | Borodin, K. and
+Turkevych, O. 1000 i 1 slovo, A1. https://doi.org/10.18452/28236 | Borodin, K.
+and Lazarenko, O. (2026). 1000 i 1 slovo, A2. https://doi.org/10.11584/opus4-1448
+| Syvokon, O. and Chaplynskyi, D. (lang-uk). ukrainian-word-stress, MIT.
+https://github.com/lang-uk/ukrainian-word-stress
+```
+
+Archived, DOI-minted copies of each dataset are listed in `sources()` under
+`conceptDoi` once they exist (today the frequency list and the stress column;
+the PULS-derived datasets wait on the profile's published license notice).
+Cite the concept DOI for the dataset you used.

uavocab-0.5.0/pyproject.toml

@@ -0,0 +1,48 @@
+[build-system]
+requires = ["setuptools>=77", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "uavocab"
+version = "0.5.0"
+description = "The open Ukrainian vocabulary data behind uavocab.org, offline: the top-12,000 lemma frequency list, the validated PULS CEFR overlay, the Borodin A1 and A2 lexical minimums, their crosswalk, the B1 enrichment tables, and a per-lemma word-stress column."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "CC-BY-NC-SA-4.0"
+license-files = ["LICENSES.md"]
+authors = [{ name = "Jake Lawrence" }]
+keywords = ["ukrainian", "uavocab", "vocabulary", "frequency list", "CEFR", "lexical minimum", "language learning", "corpus"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Education",
+  "Intended Audience :: Science/Research",
+  "Natural Language :: Ukrainian",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3 :: Only",
+  "Topic :: Education",
+  "Topic :: Text Processing :: Linguistic",
+]
+dependencies = ["pyarrow>=14"]
+
+[project.urls]
+Homepage = "https://uavocab.org/research"
+Documentation = "https://uavocab.org/research#release"
+Data = "https://uavocab.org/data"
+Source = "https://github.com/jake0lawrence/UA-Vocab"
+Issues = "https://github.com/jake0lawrence/UA-Vocab/issues"
+Changelog = "https://github.com/jake0lawrence/UA-Vocab/blob/main/python/uavocab/CHANGELOG.md"
+"Release files" = "https://github.com/jake0lawrence/UA-Vocab/tree/main/downloads/ukrainian-frequency"
+"Zenodo: frequency list" = "https://doi.org/10.5281/zenodo.22823134"
+"Zenodo: word stress" = "https://doi.org/10.5281/zenodo.23092388"
+
+[project.optional-dependencies]
+test = ["pytest>=7"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+uavocab = ["data/*.parquet", "data/manifest.json"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

uavocab-0.5.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

uavocab-0.5.0/src/uavocab/__init__.py

@@ -0,0 +1,225 @@
+"""uavocab: the open Ukrainian vocabulary data behind uavocab.org, offline.
+
+The data is the Ukrainian Frequency study at https://uavocab.org/research.
+This package was published as `ukvocab` up to 0.4.1; that name now installs
+this one.
+
+Four functions, kept small on purpose:
+
+    lookup(lemma, pos=None)      one lemma: level, its source, rank, enrichment
+    levels(lemmas)               batch: lemma -> level, with the source
+    coverage(lemmas, known_set)  the running-text coverage computation
+    sources()                    version, snapshot and license per dataset
+
+Every return value carries the dataset, version and snapshot it came from.
+The package does not lemmatize: pass lemmas in the VESUM / UberText
+convention (the infinitive for verbs, the nominative singular for nouns).
+Predicted CEFR levels are deliberately not in this package.
+"""
+
+from __future__ import annotations
+
+from collections import Counter
+from typing import Dict, Iterable, List, Optional, Union
+
+from . import _store
+from ._store import LEVELS, norm
+
+__all__ = ["lookup", "levels", "coverage", "sources", "LEVELS", "norm", "__version__"]
+
+
+def __getattr__(name):
+    if name == "__version__":
+        return _store.manifest()["packageVersion"]
+    raise AttributeError(name)
+
+
+LEVEL_SOURCE_VALIDATED = "PULS validated level, POS-aware join onto the frequency list (validated_cefr)"
+LEVEL_SOURCE_CROSSWALK = "PULS validated level via the crosswalk: lemma-string join, no part of speech (crosswalk)"
+
+
+def _reading(row: dict) -> dict:
+    lv = _store.level_by_rank().get(row["rank"])
+    return {
+        "rank": row["rank"],
+        "pos": row["pos"],
+        "count": row["count"],
+        "permille": row["permille"],
+        "cumulative_coverage": row["cumulativeCoverage"],
+        "freq_band": row["freqBand"],
+        "level": lv["level"] if lv else None,
+        "join": lv["join"] if lv else None,
+        "stress": _store.stress_by_rank().get(row["rank"]),
+    }
+
+
+def _enrichment(key: str) -> dict:
+    return {
+        "aspect": [dict(r) for r in _store.aspect_by_verb().get(key, [])],
+        "government": [dict(r) for r in _store.government_by_verb().get(key, [])],
+        "collocations": [dict(r) for r in _store.collocations_by_lemma().get(key, [])],
+    }
+
+
+def lookup(lemma: str, pos: Optional[str] = None) -> Optional[dict]:
+    """One lemma. Returns None when no table knows the string at all.
+
+    With `pos` (a UD tag such as NOUN, VERB, ADJ, DET) the reading with that
+    part of speech is returned; without it, the most frequent reading, and
+    every other reading is listed under `readings` so a homograph is never
+    silently collapsed. The level comes from the validated overlay when the
+    lemma is on the frequency list (POS-aware); otherwise from the crosswalk,
+    which joins by lemma string only, and `level_source` says which.
+    """
+    key = norm(lemma)
+    readings = _store.frequency_by_key().get(key, [])
+    if pos is not None:
+        readings = [r for r in readings if r["pos"] == pos]
+    primary = _reading(readings[0]) if readings else None
+    xw = _store.crosswalk_by_key().get(key)
+    if primary is None and xw is None:
+        return None
+
+    level = primary["level"] if primary else None
+    level_source = LEVEL_SOURCE_VALIDATED if level else None
+    if level is None and xw and xw.get("validatedLevel"):
+        level = xw["validatedLevel"]
+        level_source = LEVEL_SOURCE_CROSSWALK
+
+    used = ["frequency", "validated_cefr", "crosswalk", "borodin_a1", "borodin_a2"]
+    if primary is not None:
+        used.append("stress")
+    enrichment = _enrichment(key)
+    if any(enrichment.values()):
+        used += ["b1_aspect", "b1_government", "b1_collocations"]
+
+    return {
+        "query": lemma,
+        "lemma": readings[0]["lemma"] if readings else (xw["lemma"] if xw else key),
+        "on_list": primary is not None,
+        "rank": primary["rank"] if primary else None,
+        "pos": primary["pos"] if primary else (xw.get("pos") if xw else None),
+        "count": primary["count"] if primary else None,
+        "permille": primary["permille"] if primary else None,
+        "freq_band": primary["freq_band"] if primary else None,
+        "level": level,
+        "level_source": level_source,
+        "join": primary["join"] if primary else None,
+        "stress": primary["stress"] if primary else None,
+        "readings": [_reading(r) for r in _store.frequency_by_key().get(key, [])],
+        "minimums": {
+            "a1": key in _store.crosswalk_volume_keys("A1"),
+            "a2": key in _store.crosswalk_volume_keys("A2"),
+            "a1_entries": [r["entry"] for r in _store.borodin_keys("a1").get(key, [])],
+            "a2_entries": [r["entry"] for r in _store.borodin_keys("a2").get(key, [])],
+        },
+        "crosswalk": {
+            "class": xw["class"],
+            "volume": xw.get("volume"),
+            "validated_level": xw.get("validatedLevel"),
+            "relative": xw.get("relative"),
+            "relative_via": xw.get("relativeVia"),
+        } if xw else None,
+        "enrichment": enrichment,
+        "sources": {name: _store.source_of(name) for name in used},
+    }
+
+
+def levels(lemmas: Iterable[str]) -> Dict[str, dict]:
+    """Batch lookup: input string -> {level, level_source, rank, pos, source}.
+    A lemma no table knows gets level None and an empty source."""
+    out: Dict[str, dict] = {}
+    for lemma in lemmas:
+        hit = lookup(lemma)
+        if hit is None:
+            out[lemma] = {"level": None, "level_source": None, "rank": None, "pos": None, "source": None}
+            continue
+        src_name = "validated_cefr" if hit["level_source"] == LEVEL_SOURCE_VALIDATED else ("crosswalk" if hit["level"] else None)
+        out[lemma] = {
+            "level": hit["level"],
+            "level_source": hit["level_source"],
+            "rank": hit["rank"],
+            "pos": hit["pos"],
+            "source": _store.source_of(src_name) if src_name else None,
+        }
+    return out
+
+
+KnownSet = Union[str, int, Iterable[str]]
+
+
+def _resolve_known(known_set: KnownSet):
+    """A known set is an iterable of lemma strings, a level name (every
+    validated lemma at or below it), 'borodin-a1' / 'borodin-a2' (the
+    minimum, on and off the list, A2 including A1), or an int N (the top N of
+    the frequency list)."""
+    if isinstance(known_set, int) and not isinstance(known_set, bool):
+        return _store.top_n_set(known_set), f"top {known_set} of the frequency list", ["frequency"]
+    if isinstance(known_set, str):
+        tag = known_set.strip()
+        if tag.upper() in LEVELS:
+            return _store.level_set(tag.upper()), f"every validated lemma at or below {tag.upper()}", ["validated_cefr", "frequency"]
+        if tag.lower() in ("borodin-a1", "a1-minimum"):
+            return _store.crosswalk_volume_keys("A1"), "the Borodin A1 minimum (every headword, on or off the list)", ["crosswalk", "borodin_a1"]
+        if tag.lower() in ("borodin-a2", "a2-minimum"):
+            return _store.crosswalk_volume_keys("A2"), "the Borodin A1 and A2 minimums (every headword, on or off the list)", ["crosswalk", "borodin_a1", "borodin_a2"]
+        raise ValueError(f"unknown known_set name {known_set!r}: a level (A1..C1), 'borodin-a1', 'borodin-a2', an int, or an iterable of lemmas")
+    return frozenset(norm(s) for s in known_set), "a caller-supplied set of lemmas", ["frequency"]
+
+
+def coverage(lemmas: Iterable[str], known_set: KnownSet, top_unknown: int = 20) -> dict:
+    """Running-text coverage of a lemmatized text against a known set.
+
+    `lemmas` is the text as a sequence of lemmas, one per token, repeats
+    included (coverage is token coverage, not type coverage). Tokens are
+    counted as known when their normalized lemma is in the known set; the
+    package does not decide what a word is, so pass only the tokens you want
+    counted (a caller usually drops punctuation, numbers and proper names).
+    Also reports how many tokens are on the frequency list at all, and the
+    most frequent unknown lemmas, so a gap is a list rather than a number.
+    """
+    known, description, used = _resolve_known(known_set)
+    counts = Counter(norm(t) for t in lemmas)
+    total = sum(counts.values())
+    known_tokens = sum(n for k, n in counts.items() if k in known)
+    on_list = sum(n for k, n in counts.items() if k in _store.frequency_by_key())
+    unknown = Counter({k: n for k, n in counts.items() if k not in known})
+    by_level: Dict[str, int] = {lvl: 0 for lvl in LEVELS}
+    unleveled = 0
+    for k, n in counts.items():
+        readings = _store.frequency_by_key().get(k)
+        lv = _store.level_by_rank().get(readings[0]["rank"]) if readings else None
+        if lv:
+            by_level[lv["level"]] += n
+        else:
+            unleveled += n
+    return {
+        "tokens": total,
+        "types": len(counts),
+        "known": known_tokens,
+        "unknown": total - known_tokens,
+        "coverage": (known_tokens / total) if total else 0.0,
+        "on_list": on_list,
+        "on_list_share": (on_list / total) if total else 0.0,
+        "by_validated_level": by_level,
+        "unleveled": unleveled,
+        "known_set": description,
+        "known_set_size": len(known),
+        "top_unknown": unknown.most_common(top_unknown),
+        "sources": {name: _store.source_of(name) for name in used},
+    }
+
+
+def sources() -> dict:
+    """Version, snapshot, license, attribution, citation and checksum per
+    dataset, plus the combined license and why, straight from the manifest."""
+    m = _store.manifest()
+    return {
+        "package": m["package"],
+        "package_version": m["packageVersion"],
+        "site": m["site"],
+        "combined_license": m["combinedLicense"],
+        "combined_license_why": m["combinedLicenseWhy"],
+        "not_shipped": list(m["notShipped"]),
+        "datasets": {name: dict(d) for name, d in m["datasets"].items()},
+    }