ukvocab 0.3.0__tar.gz

ukvocab-0.3.0/LICENSES.md

@@ -0,0 +1,49 @@
+# Licenses
+
+## The combined work: CC-BY-NC-SA 4.0
+
+This package as a whole is distributed under the Creative Commons
+Attribution-NonCommercial-ShareAlike 4.0 International license
+(https://creativecommons.org/licenses/by-nc-sa/4.0/).
+
+Why NonCommercial and share-alike: the per-lemma CEFR levels come from the
+Ukrainian Vocabulary Profile (PULS), which its authors license CC BY-NC-SA 4.0
+(decided 2026-09-24) and which is redistributed here with their permission.
+Those levels are in the validated overlay, in every row of the crosswalk, in
+the `puls_level` column of the two Borodin tables, and in the `related_level`
+column of the aspect table. NonCommercial and share-alike propagate to a
+combined work, so the package carries both. They are the one NonCommercial
+component; nothing else NonCommercial is shipped. The frequency list and the
+two Borodin minimums are CC-BY 4.0 on their own and remain so, and the
+government and collocation tables are CC-BY-SA 4.0; each dataset's own license
+is recorded in `ukvocab.sources()` and below, so a downstream user who drops
+the PULS level columns owes attribution (and, for the corpus tables,
+share-alike) and nothing more.
+
+Cite the profile as its authors ask: Профіль української лексики (ПУЛЬС) / О. Синчак, В. Старко, М. Бурак, М. Свистун та ін. Львів: Школа української мови та культури УКУ, 2026.
+
+## Per dataset
+
+| Dataset | License | Attribution |
+|---|---|---|
+| `frequency` (top-12,000 lemma list) | CC-BY 4.0 | Derived aggregate counts from the UberText 2.0 lemma frequency dictionary, lang-uk. Chaplynskyi, D. (2023). Introducing UberText 2.0: A Corpus of Modern Ukrainian at Scale. Proc. 2nd UNLP, ACL. https://lang.org.ua/en/ubertext/ |
+| `validated_cefr` (PULS CEFR overlay) | CC-BY-NC-SA 4.0 | Ukrainian Vocabulary Profile (PULS): Synchak, O., Starko, V., Burak, M., Svystun, M. https://puls.peremova.org/ (eLex 2025). Per-lemma levels redistributed with the authors' permission (V. Starko, July 2026); CC BY-NC-SA 4.0 at the authors' decision (O. Synchak, 2026-09-24), attribution required. Snapshot 2026-07-23; the profile is not final. |
+| `borodin_a1` (A1 lexical minimum) | CC-BY 4.0 | Borodin, K. and Turkevych, O. 1000 i 1 slovo. Leksychnyi minimum z ukrainskoi movy yak inozemnoi. Riven A1. Slavic Language Education 3, LANGUAGE-LAB, Humboldt-Universitat zu Berlin. https://doi.org/10.18452/28236. Machine-readable list provided by the authors, 2026-07-27. The `puls_level` column is CC-BY-NC-SA (PULS). |
+| `borodin_a2` (A2 lexical minimum) | CC-BY 4.0 | Borodin, K. and Lazarenko, O. (2026). 1000 i 1 slovo. Riven A2. Europa-Universitat Viadrina Frankfurt (Oder). https://doi.org/10.11584/opus4-1448. The `puls_level` column is CC-BY-NC-SA (PULS). |
+| `crosswalk` (minimums against the profile) | CC-BY-NC-SA 4.0 | Lawrence, J. (2026), over the two Borodin volumes (CC-BY 4.0), the PULS profile (CC BY-NC-SA 4.0, by permission) and UberText 2.0 ranks (CC-BY 4.0). |
+| `b1_aspect` (aspect relations) | CC-BY-NC-SA 4.0 | Levels: PULS, by permission, CC BY-NC-SA 4.0. Aspect partners: Wiktionary via kaikki.org, CC-BY-SA 3.0. Counts: UberText 2.0, CC-BY 4.0. |
+| `b1_government` (case government) | CC-BY-SA 4.0 | Counts over the UberText 2.0 Wikipedia subcorpus (lang-uk, CC-BY 4.0); derived statistics CC-BY-SA 4.0. |
+| `b1_collocations` (logDice collocations) | CC-BY-SA 4.0 | Same corpus and terms as `b1_government`. |
+
+## Not in this package
+
+Predicted CEFR levels (the site's gap-fill model; predictions are never merged
+into the validated overlay), any VESUM-derived lexicon or wordform data
+(CC BY-NC-SA), the UniMorph derivational families (CC BY-NC 4.0), the
+Wikipedia example sentences, and the Wiktionary paradigm tables. If you need
+any of them, they are on the site with their own terms.
+
+## The code
+
+The Python code in this package (`src/ukvocab/*.py`) is released under the
+same CC-BY-NC-SA 4.0 terms as the data, to keep one license on the wheel.

ukvocab-0.3.0/PKG-INFO

@@ -0,0 +1,99 @@
+Metadata-Version: 2.4
+Name: ukvocab
+Version: 0.3.0
+Summary: The published Ukrainian vocabulary data from jakelawrence.xyz/research/ukrainian-frequency, offline: the top-12,000 lemma frequency list, the validated PULS CEFR overlay, the Borodin A1 and A2 lexical minimums, their crosswalk, and the B1 enrichment tables.
+Author: Jake Lawrence
+License-Expression: CC-BY-NC-SA-4.0
+Project-URL: Homepage, https://www.jakelawrence.xyz/research/ukrainian-frequency
+Project-URL: Source, https://github.com/jake0lawrence/jakelawrence.xyz/tree/main/packages/ukvocab
+Project-URL: Data, https://www.jakelawrence.xyz/downloads/ukrainian-frequency/
+Keywords: ukrainian,vocabulary,frequency list,CEFR,lexical minimum,language learning,corpus
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: Science/Research
+Classifier: Natural Language :: Ukrainian
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Education
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSES.md
+Requires-Dist: pyarrow>=14
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Dynamic: license-file
+
+# ukvocab
+
+The published Ukrainian vocabulary data from
+[jakelawrence.xyz/research/ukrainian-frequency](https://www.jakelawrence.xyz/research/ukrainian-frequency),
+installable and offline: the top-12,000 lemma frequency list over UberText 2.0,
+the validated PULS CEFR overlay (A1 to C1), the Borodin A1 and A2 lexical
+minimums, the crosswalk between the minimums and the profile, and the B1
+enrichment tables (aspect relations, case government, collocations).
+
+```python
+import ukvocab
+ukvocab.lookup("п'ять")["level"]                       # 'A1', with rank, source and enrichment beside it
+ukvocab.coverage(["в", "школа", "йти"], "A2")["coverage"]  # share of tokens a validated-A2 learner knows
+```
+
+## What you get
+
+| function | returns |
+|---|---|
+| `lookup(lemma, pos=None)` | one lemma: `level`, `level_source`, `rank`, the frequency figures, every other reading of the spelling, minimum membership, the crosswalk class, and the B1 `enrichment` (aspect partners, government, collocations); `None` when no table knows the string |
+| `levels(lemmas)` | batch: input string to `{level, level_source, rank, pos, source}` |
+| `coverage(lemmas, known_set)` | token coverage of a lemmatized text against a known set: a level (`"A2"` means every validated lemma at or below A2), `"borodin-a1"` / `"borodin-a2"`, an int N (top N of the list), or your own set of lemmas; reports the unknowns by frequency |
+| `sources()` | version, snapshot date, license, attribution, citation and SHA-256 per dataset, the combined license and why, and what is deliberately not shipped |
+
+Every return value carries the dataset, version and snapshot it came from.
+Data is Parquet, loaded lazily on first use, never on import. The only
+dependency is `pyarrow`.
+
+Two things the package does not do, on purpose. It does not lemmatize: pass
+lemmas in the UberText / VESUM convention (infinitive for verbs, nominative
+singular for nouns); the site's `POST /api/lemmatize` is the hosted lemmatizer
+matched to this list. And it ships no predicted CEFR levels: only the profile's
+validated levels are here, and a lemma the profile does not cover gets
+`level: None` rather than a guess.
+
+## Provenance and honesty
+
+The data tier is generated from the site's published download files by
+`scripts/ukrainian-frequency/build-package.py` in the source repository and is
+never edited by hand; the tests assert that every table's row count and hash
+match the published file exactly. `sources()` is the manifest that build
+wrote. The PULS overlay is a dated snapshot (2026-07-23) of a profile that is
+still being extended, redistributed with its authors' permission. The join
+onto the frequency list is part-of-speech aware; ten declared cross-tag rules
+recover the words the two tag conventions merely label differently, and each
+row's `join` field says how it matched, so the strict join is one filter away.
+
+## License
+
+The combined package is **CC-BY-NC-SA 4.0**, because the PULS levels are
+NonCommercial and share-alike at the authors' decision (2026-09-24) and they run
+through most tables. The frequency list and the two Borodin minimums are CC-BY
+4.0 on their own, and the government and collocation tables CC-BY-SA 4.0; drop
+the PULS level columns and NonCommercial does not apply. `LICENSES.md` has the
+per-dataset terms and every attribution.
+
+## Citation
+
+```
+Lawrence, J. (2026). ukvocab: the Ukrainian Frequency datasets (top-12,000 lemma
+list, validated PULS CEFR overlay, Borodin A1/A2 minimums, crosswalk, B1
+enrichment tables), version 0.2.0. jakelawrence.xyz/research/ukrainian-frequency.
+
+Upstream: Chaplynskyi, D. (2023). Introducing UberText 2.0: A Corpus of Modern
+Ukrainian at Scale. Proc. 2nd UNLP, ACL. | Synchak, O., Starko, V., Burak, M.,
+Svystun, M. (2025). Corpus-Based Vocabulary Profiling for Ukrainian: From
+Lexical Analysis to CEFR. eLex 2025; PULS, puls.peremova.org. | Borodin, K. and
+Turkevych, O. 1000 i 1 slovo, A1. https://doi.org/10.18452/28236 | Borodin, K.
+and Lazarenko, O. (2026). 1000 i 1 slovo, A2. https://doi.org/10.11584/opus4-1448
+```
+
+Archived, DOI-minted copies of each dataset are listed in `sources()` under
+`conceptDoi` once they exist; cite the concept DOI for the dataset you used.

ukvocab-0.3.0/README.md

@@ -0,0 +1,73 @@
+# ukvocab
+
+The published Ukrainian vocabulary data from
+[jakelawrence.xyz/research/ukrainian-frequency](https://www.jakelawrence.xyz/research/ukrainian-frequency),
+installable and offline: the top-12,000 lemma frequency list over UberText 2.0,
+the validated PULS CEFR overlay (A1 to C1), the Borodin A1 and A2 lexical
+minimums, the crosswalk between the minimums and the profile, and the B1
+enrichment tables (aspect relations, case government, collocations).
+
+```python
+import ukvocab
+ukvocab.lookup("п'ять")["level"]                       # 'A1', with rank, source and enrichment beside it
+ukvocab.coverage(["в", "школа", "йти"], "A2")["coverage"]  # share of tokens a validated-A2 learner knows
+```
+
+## What you get
+
+| function | returns |
+|---|---|
+| `lookup(lemma, pos=None)` | one lemma: `level`, `level_source`, `rank`, the frequency figures, every other reading of the spelling, minimum membership, the crosswalk class, and the B1 `enrichment` (aspect partners, government, collocations); `None` when no table knows the string |
+| `levels(lemmas)` | batch: input string to `{level, level_source, rank, pos, source}` |
+| `coverage(lemmas, known_set)` | token coverage of a lemmatized text against a known set: a level (`"A2"` means every validated lemma at or below A2), `"borodin-a1"` / `"borodin-a2"`, an int N (top N of the list), or your own set of lemmas; reports the unknowns by frequency |
+| `sources()` | version, snapshot date, license, attribution, citation and SHA-256 per dataset, the combined license and why, and what is deliberately not shipped |
+
+Every return value carries the dataset, version and snapshot it came from.
+Data is Parquet, loaded lazily on first use, never on import. The only
+dependency is `pyarrow`.
+
+Two things the package does not do, on purpose. It does not lemmatize: pass
+lemmas in the UberText / VESUM convention (infinitive for verbs, nominative
+singular for nouns); the site's `POST /api/lemmatize` is the hosted lemmatizer
+matched to this list. And it ships no predicted CEFR levels: only the profile's
+validated levels are here, and a lemma the profile does not cover gets
+`level: None` rather than a guess.
+
+## Provenance and honesty
+
+The data tier is generated from the site's published download files by
+`scripts/ukrainian-frequency/build-package.py` in the source repository and is
+never edited by hand; the tests assert that every table's row count and hash
+match the published file exactly. `sources()` is the manifest that build
+wrote. The PULS overlay is a dated snapshot (2026-07-23) of a profile that is
+still being extended, redistributed with its authors' permission. The join
+onto the frequency list is part-of-speech aware; ten declared cross-tag rules
+recover the words the two tag conventions merely label differently, and each
+row's `join` field says how it matched, so the strict join is one filter away.
+
+## License
+
+The combined package is **CC-BY-NC-SA 4.0**, because the PULS levels are
+NonCommercial and share-alike at the authors' decision (2026-09-24) and they run
+through most tables. The frequency list and the two Borodin minimums are CC-BY
+4.0 on their own, and the government and collocation tables CC-BY-SA 4.0; drop
+the PULS level columns and NonCommercial does not apply. `LICENSES.md` has the
+per-dataset terms and every attribution.
+
+## Citation
+
+```
+Lawrence, J. (2026). ukvocab: the Ukrainian Frequency datasets (top-12,000 lemma
+list, validated PULS CEFR overlay, Borodin A1/A2 minimums, crosswalk, B1
+enrichment tables), version 0.2.0. jakelawrence.xyz/research/ukrainian-frequency.
+
+Upstream: Chaplynskyi, D. (2023). Introducing UberText 2.0: A Corpus of Modern
+Ukrainian at Scale. Proc. 2nd UNLP, ACL. | Synchak, O., Starko, V., Burak, M.,
+Svystun, M. (2025). Corpus-Based Vocabulary Profiling for Ukrainian: From
+Lexical Analysis to CEFR. eLex 2025; PULS, puls.peremova.org. | Borodin, K. and
+Turkevych, O. 1000 i 1 slovo, A1. https://doi.org/10.18452/28236 | Borodin, K.
+and Lazarenko, O. (2026). 1000 i 1 slovo, A2. https://doi.org/10.11584/opus4-1448
+```
+
+Archived, DOI-minted copies of each dataset are listed in `sources()` under
+`conceptDoi` once they exist; cite the concept DOI for the dataset you used.

ukvocab-0.3.0/pyproject.toml

@@ -0,0 +1,42 @@
+[build-system]
+requires = ["setuptools>=77", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ukvocab"
+version = "0.3.0"
+description = "The published Ukrainian vocabulary data from jakelawrence.xyz/research/ukrainian-frequency, offline: the top-12,000 lemma frequency list, the validated PULS CEFR overlay, the Borodin A1 and A2 lexical minimums, their crosswalk, and the B1 enrichment tables."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "CC-BY-NC-SA-4.0"
+license-files = ["LICENSES.md"]
+authors = [{ name = "Jake Lawrence" }]
+keywords = ["ukrainian", "vocabulary", "frequency list", "CEFR", "lexical minimum", "language learning", "corpus"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Education",
+  "Intended Audience :: Science/Research",
+  "Natural Language :: Ukrainian",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3 :: Only",
+  "Topic :: Education",
+  "Topic :: Text Processing :: Linguistic",
+]
+dependencies = ["pyarrow>=14"]
+
+[project.urls]
+Homepage = "https://www.jakelawrence.xyz/research/ukrainian-frequency"
+Source = "https://github.com/jake0lawrence/jakelawrence.xyz/tree/main/packages/ukvocab"
+Data = "https://www.jakelawrence.xyz/downloads/ukrainian-frequency/"
+
+[project.optional-dependencies]
+test = ["pytest>=7"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+ukvocab = ["data/*.parquet", "data/manifest.json"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

ukvocab-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

ukvocab-0.3.0/src/ukvocab/__init__.py

@@ -0,0 +1,218 @@
+"""ukvocab: the published Ukrainian vocabulary data from
+jakelawrence.xyz/research/ukrainian-frequency, offline.
+
+Four functions, kept small on purpose:
+
+    lookup(lemma, pos=None)      one lemma: level, its source, rank, enrichment
+    levels(lemmas)               batch: lemma -> level, with the source
+    coverage(lemmas, known_set)  the running-text coverage computation
+    sources()                    version, snapshot and license per dataset
+
+Every return value carries the dataset, version and snapshot it came from.
+The package does not lemmatize: pass lemmas in the VESUM / UberText
+convention (the infinitive for verbs, the nominative singular for nouns).
+Predicted CEFR levels are deliberately not in this package.
+"""
+
+from __future__ import annotations
+
+from collections import Counter
+from typing import Dict, Iterable, List, Optional, Union
+
+from . import _store
+from ._store import LEVELS, norm
+
+__all__ = ["lookup", "levels", "coverage", "sources", "LEVELS", "norm", "__version__"]
+
+
+def __getattr__(name):
+    if name == "__version__":
+        return _store.manifest()["packageVersion"]
+    raise AttributeError(name)
+
+
+LEVEL_SOURCE_VALIDATED = "PULS validated level, POS-aware join onto the frequency list (validated_cefr)"
+LEVEL_SOURCE_CROSSWALK = "PULS validated level via the crosswalk: lemma-string join, no part of speech (crosswalk)"
+
+
+def _reading(row: dict) -> dict:
+    lv = _store.level_by_rank().get(row["rank"])
+    return {
+        "rank": row["rank"],
+        "pos": row["pos"],
+        "count": row["count"],
+        "permille": row["permille"],
+        "cumulative_coverage": row["cumulativeCoverage"],
+        "freq_band": row["freqBand"],
+        "level": lv["level"] if lv else None,
+        "join": lv["join"] if lv else None,
+    }
+
+
+def _enrichment(key: str) -> dict:
+    return {
+        "aspect": [dict(r) for r in _store.aspect_by_verb().get(key, [])],
+        "government": [dict(r) for r in _store.government_by_verb().get(key, [])],
+        "collocations": [dict(r) for r in _store.collocations_by_lemma().get(key, [])],
+    }
+
+
+def lookup(lemma: str, pos: Optional[str] = None) -> Optional[dict]:
+    """One lemma. Returns None when no table knows the string at all.
+
+    With `pos` (a UD tag such as NOUN, VERB, ADJ, DET) the reading with that
+    part of speech is returned; without it, the most frequent reading, and
+    every other reading is listed under `readings` so a homograph is never
+    silently collapsed. The level comes from the validated overlay when the
+    lemma is on the frequency list (POS-aware); otherwise from the crosswalk,
+    which joins by lemma string only, and `level_source` says which.
+    """
+    key = norm(lemma)
+    readings = _store.frequency_by_key().get(key, [])
+    if pos is not None:
+        readings = [r for r in readings if r["pos"] == pos]
+    primary = _reading(readings[0]) if readings else None
+    xw = _store.crosswalk_by_key().get(key)
+    if primary is None and xw is None:
+        return None
+
+    level = primary["level"] if primary else None
+    level_source = LEVEL_SOURCE_VALIDATED if level else None
+    if level is None and xw and xw.get("validatedLevel"):
+        level = xw["validatedLevel"]
+        level_source = LEVEL_SOURCE_CROSSWALK
+
+    used = ["frequency", "validated_cefr", "crosswalk", "borodin_a1", "borodin_a2"]
+    enrichment = _enrichment(key)
+    if any(enrichment.values()):
+        used += ["b1_aspect", "b1_government", "b1_collocations"]
+
+    return {
+        "query": lemma,
+        "lemma": readings[0]["lemma"] if readings else (xw["lemma"] if xw else key),
+        "on_list": primary is not None,
+        "rank": primary["rank"] if primary else None,
+        "pos": primary["pos"] if primary else (xw.get("pos") if xw else None),
+        "count": primary["count"] if primary else None,
+        "permille": primary["permille"] if primary else None,
+        "freq_band": primary["freq_band"] if primary else None,
+        "level": level,
+        "level_source": level_source,
+        "join": primary["join"] if primary else None,
+        "readings": [_reading(r) for r in _store.frequency_by_key().get(key, [])],
+        "minimums": {
+            "a1": key in _store.crosswalk_volume_keys("A1"),
+            "a2": key in _store.crosswalk_volume_keys("A2"),
+            "a1_entries": [r["entry"] for r in _store.borodin_keys("a1").get(key, [])],
+            "a2_entries": [r["entry"] for r in _store.borodin_keys("a2").get(key, [])],
+        },
+        "crosswalk": {
+            "class": xw["class"],
+            "volume": xw.get("volume"),
+            "validated_level": xw.get("validatedLevel"),
+            "relative": xw.get("relative"),
+            "relative_via": xw.get("relativeVia"),
+        } if xw else None,
+        "enrichment": enrichment,
+        "sources": {name: _store.source_of(name) for name in used},
+    }
+
+
+def levels(lemmas: Iterable[str]) -> Dict[str, dict]:
+    """Batch lookup: input string -> {level, level_source, rank, pos, source}.
+    A lemma no table knows gets level None and an empty source."""
+    out: Dict[str, dict] = {}
+    for lemma in lemmas:
+        hit = lookup(lemma)
+        if hit is None:
+            out[lemma] = {"level": None, "level_source": None, "rank": None, "pos": None, "source": None}
+            continue
+        src_name = "validated_cefr" if hit["level_source"] == LEVEL_SOURCE_VALIDATED else ("crosswalk" if hit["level"] else None)
+        out[lemma] = {
+            "level": hit["level"],
+            "level_source": hit["level_source"],
+            "rank": hit["rank"],
+            "pos": hit["pos"],
+            "source": _store.source_of(src_name) if src_name else None,
+        }
+    return out
+
+
+KnownSet = Union[str, int, Iterable[str]]
+
+
+def _resolve_known(known_set: KnownSet):
+    """A known set is an iterable of lemma strings, a level name (every
+    validated lemma at or below it), 'borodin-a1' / 'borodin-a2' (the
+    minimum, on and off the list, A2 including A1), or an int N (the top N of
+    the frequency list)."""
+    if isinstance(known_set, int) and not isinstance(known_set, bool):
+        return _store.top_n_set(known_set), f"top {known_set} of the frequency list", ["frequency"]
+    if isinstance(known_set, str):
+        tag = known_set.strip()
+        if tag.upper() in LEVELS:
+            return _store.level_set(tag.upper()), f"every validated lemma at or below {tag.upper()}", ["validated_cefr", "frequency"]
+        if tag.lower() in ("borodin-a1", "a1-minimum"):
+            return _store.crosswalk_volume_keys("A1"), "the Borodin A1 minimum (every headword, on or off the list)", ["crosswalk", "borodin_a1"]
+        if tag.lower() in ("borodin-a2", "a2-minimum"):
+            return _store.crosswalk_volume_keys("A2"), "the Borodin A1 and A2 minimums (every headword, on or off the list)", ["crosswalk", "borodin_a1", "borodin_a2"]
+        raise ValueError(f"unknown known_set name {known_set!r}: a level (A1..C1), 'borodin-a1', 'borodin-a2', an int, or an iterable of lemmas")
+    return frozenset(norm(s) for s in known_set), "a caller-supplied set of lemmas", ["frequency"]
+
+
+def coverage(lemmas: Iterable[str], known_set: KnownSet, top_unknown: int = 20) -> dict:
+    """Running-text coverage of a lemmatized text against a known set.
+
+    `lemmas` is the text as a sequence of lemmas, one per token, repeats
+    included (coverage is token coverage, not type coverage). Tokens are
+    counted as known when their normalized lemma is in the known set; the
+    package does not decide what a word is, so pass only the tokens you want
+    counted (a caller usually drops punctuation, numbers and proper names).
+    Also reports how many tokens are on the frequency list at all, and the
+    most frequent unknown lemmas, so a gap is a list rather than a number.
+    """
+    known, description, used = _resolve_known(known_set)
+    counts = Counter(norm(t) for t in lemmas)
+    total = sum(counts.values())
+    known_tokens = sum(n for k, n in counts.items() if k in known)
+    on_list = sum(n for k, n in counts.items() if k in _store.frequency_by_key())
+    unknown = Counter({k: n for k, n in counts.items() if k not in known})
+    by_level: Dict[str, int] = {lvl: 0 for lvl in LEVELS}
+    unleveled = 0
+    for k, n in counts.items():
+        readings = _store.frequency_by_key().get(k)
+        lv = _store.level_by_rank().get(readings[0]["rank"]) if readings else None
+        if lv:
+            by_level[lv["level"]] += n
+        else:
+            unleveled += n
+    return {
+        "tokens": total,
+        "types": len(counts),
+        "known": known_tokens,
+        "unknown": total - known_tokens,
+        "coverage": (known_tokens / total) if total else 0.0,
+        "on_list": on_list,
+        "on_list_share": (on_list / total) if total else 0.0,
+        "by_validated_level": by_level,
+        "unleveled": unleveled,
+        "known_set": description,
+        "known_set_size": len(known),
+        "top_unknown": unknown.most_common(top_unknown),
+        "sources": {name: _store.source_of(name) for name in used},
+    }
+
+
+def sources() -> dict:
+    """Version, snapshot, license, attribution, citation and checksum per
+    dataset, plus the combined license and why, straight from the manifest."""
+    m = _store.manifest()
+    return {
+        "package": m["package"],
+        "package_version": m["packageVersion"],
+        "site": m["site"],
+        "combined_license": m["combinedLicense"],
+        "combined_license_why": m["combinedLicenseWhy"],
+        "not_shipped": list(m["notShipped"]),
+        "datasets": {name: dict(d) for name, d in m["datasets"].items()},
+    }

ukvocab-0.3.0/src/ukvocab/_store.py

@@ -0,0 +1,172 @@
+"""Lazy, read-only access to the package's Parquet data tier.
+
+Nothing is read at import time. Each table loads on first use through
+pyarrow, and the lookup indexes are built once per process. The manifest
+(manifest.json, generated by scripts/ukrainian-frequency/build-package.py in
+the source repository) is the only description of the data; this module
+never hard-codes a row count, a license or a snapshot date.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import unicodedata
+from functools import lru_cache
+from typing import Dict, List, Optional
+
+DATA_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "data")
+
+LEVELS = ("A1", "A2", "B1", "B2", "C1")
+LEVEL_INDEX = {lvl: i for i, lvl in enumerate(LEVELS)}
+
+_APOSTROPHES = ("'", "ʼ", "’", "‘", "ʹ", "`", "´")
+
+
+def norm(s: str) -> str:
+    """The join key used across every table: NFC, the pedagogical stress mark
+    (U+0301) stripped, every apostrophe glyph folded to U+2019 (the frequency
+    list's glyph), whitespace trimmed, lowercased."""
+    s = unicodedata.normalize("NFD", s or "")
+    s = "".join(ch for ch in s if ch != "́")
+    s = unicodedata.normalize("NFC", s)
+    for glyph in _APOSTROPHES:
+        s = s.replace(glyph, "’")
+    return s.strip().lower()
+
+
+@lru_cache(maxsize=None)
+def manifest() -> dict:
+    with open(os.path.join(DATA_DIR, "manifest.json"), encoding="utf-8") as fh:
+        return json.load(fh)
+
+
+@lru_cache(maxsize=None)
+def table(name: str):
+    """The pyarrow Table for a dataset name from the manifest."""
+    import pyarrow.parquet as pq
+
+    meta = manifest()["datasets"][name]
+    return pq.read_table(os.path.join(DATA_DIR, meta["file"]))
+
+
+@lru_cache(maxsize=None)
+def rows(name: str) -> List[dict]:
+    """The dataset as a list of plain dicts (nulls as None)."""
+    return table(name).to_pylist()
+
+
+def source_of(name: str) -> dict:
+    """The provenance stamp every return value carries for a dataset."""
+    m = manifest()["datasets"][name]
+    return {
+        "dataset": name,
+        "version": m["version"],
+        "snapshot": m["snapshot"],
+        "license": m["license"],
+        "rows": m["rows"],
+        "sha256": m["sha256"],
+        "conceptDoi": m.get("conceptDoi"),
+    }
+
+
+# ---------------------------------------------------------------- indexes
+@lru_cache(maxsize=None)
+def frequency_by_key() -> Dict[str, List[dict]]:
+    """norm(lemma) -> frequency rows (rank order); a spelling can have several
+    readings because the corpus tagger split it by part of speech."""
+    out: Dict[str, List[dict]] = {}
+    for r in rows("frequency"):
+        out.setdefault(norm(r["lemma"]), []).append(r)
+    for v in out.values():
+        v.sort(key=lambda r: r["rank"])
+    return out
+
+
+@lru_cache(maxsize=None)
+def frequency_by_rank() -> Dict[int, dict]:
+    return {r["rank"]: r for r in rows("frequency")}
+
+
+@lru_cache(maxsize=None)
+def level_by_rank() -> Dict[int, dict]:
+    """rank -> validated overlay row (level, join)."""
+    return {r["rank"]: r for r in rows("validated_cefr")}
+
+
+@lru_cache(maxsize=None)
+def crosswalk_by_key() -> Dict[str, dict]:
+    return {norm(r["lemma"]): r for r in rows("crosswalk")}
+
+
+@lru_cache(maxsize=None)
+def borodin_keys(volume: str) -> Dict[str, List[dict]]:
+    """norm(headword lemma) -> the matched entries of a Borodin volume. Only
+    matched entries carry a lemma; off-list headwords are reached through the
+    crosswalk's volume column instead."""
+    name = {"a1": "borodin_a1", "a2": "borodin_a2"}[volume]
+    out: Dict[str, List[dict]] = {}
+    for r in rows(name):
+        if r.get("matched") and r.get("lemma"):
+            out.setdefault(norm(r["lemma"]), []).append(r)
+    return out
+
+
+@lru_cache(maxsize=None)
+def crosswalk_volume_keys(volume: str) -> frozenset:
+    """Every lemma string a Borodin volume carries, on or off the frequency
+    list, as the crosswalk records it (a word in both volumes counts as A1)."""
+    keys = set()
+    for r in rows("crosswalk"):
+        v = r.get("volume")
+        if v == volume or (volume == "A2" and v == "A1"):
+            keys.add(norm(r["lemma"]))
+    return frozenset(keys)
+
+
+@lru_cache(maxsize=None)
+def aspect_by_verb() -> Dict[str, List[dict]]:
+    out: Dict[str, List[dict]] = {}
+    for r in rows("b1_aspect"):
+        out.setdefault(norm(r["b1_verb"]), []).append(r)
+    return out
+
+
+@lru_cache(maxsize=None)
+def government_by_verb() -> Dict[str, List[dict]]:
+    out: Dict[str, List[dict]] = {}
+    for r in rows("b1_government"):
+        out.setdefault(norm(r["verb"]), []).append(r)
+    return out
+
+
+@lru_cache(maxsize=None)
+def collocations_by_lemma() -> Dict[str, List[dict]]:
+    out: Dict[str, List[dict]] = {}
+    for r in rows("b1_collocations"):
+        out.setdefault(norm(r["lemma"]), []).append(r)
+    return out
+
+
+@lru_cache(maxsize=None)
+def level_set(level: str) -> frozenset:
+    """norm(lemma) of every validated lemma at or below a level."""
+    if level not in LEVEL_INDEX:
+        raise ValueError(f"unknown level {level!r}; one of {LEVELS}")
+    top = LEVEL_INDEX[level]
+    by_rank = frequency_by_rank()
+    return frozenset(norm(by_rank[r["rank"]]["lemma"]) for r in rows("validated_cefr") if LEVEL_INDEX[r["level"]] <= top)
+
+
+@lru_cache(maxsize=None)
+def top_n_set(n: int) -> frozenset:
+    return frozenset(norm(r["lemma"]) for r in rows("frequency") if r["rank"] <= n)
+
+
+def clear_cache() -> None:
+    """Drop every loaded table and index (tests, or long processes that want
+    the memory back)."""
+    for fn in (manifest, table, rows, frequency_by_key, frequency_by_rank, level_by_rank, crosswalk_by_key,
+               borodin_keys, crosswalk_volume_keys, aspect_by_verb, government_by_verb, collocations_by_lemma,
+               level_set, top_n_set):
+        fn.cache_clear()