acore-utils 0.1.2__py3-none-any.whl

acore/__init__.py

@@ -0,0 +1,7 @@
+"""acore: Agent-based Comparative Reconstruction library."""
+
+__version__ = "0.1.2"
+
+from acore import ipa, pi, rho
+
+__all__ = ["ipa", "pi", "rho"]

acore/cldf_utils.md

@@ -0,0 +1,98 @@
+# cldf_utils
+
+Functions that take a CLDF comparative dictionary from raw data to predicted protoforms, and print each step as a table.
+
+```python
+import cldf_utils as utils
+
+ta = utils.load_cldf("../../cldf-data/comparative-tangkhulic-cldf")  # 0. read the data
+cogs = utils.cognates(ta, proto="17", langs=["4", "1", "2", "3"])     # 1. pick cognate sets
+aligned = utils.align_cognates(cogs)                                  # 2. line up the sounds
+rules = utils.correspond(aligned)                                     # 3. find rules
+predicted = utils.predict(rules, aligned)                             # 4. reconstruct
+
+utils.print_cognates(cogs[:10])                                       # show any step
+```
+
+Every step returns a list in the same order: `aligned[i]` and `predicted[i]` belong to `cogs[i]`.
+
+Languages are identified by their **CLDF ID**, the `ID` column of `languages.csv`. These are numbers in Austronesian and Tangkhulic (`"4"` is Ukhrul), and names in Uralic (`"Finnish"`) and Chinese (`"Cantonese"`). `ta["languages"]` lists each ID with its name.
+
+Words are split into sounds by [PanPhon](https://github.com/dmort27/panphon)'s `ipa_segs`, and sets are aligned by [emalign](https://github.com/changelinglab/emalign). Both are in the project's `cldf-utils` dependency group, so from the repository root:
+
+```bash
+uv sync --group cldf-utils           # or: pip install panphon "emalign @ git+https://github.com/changelinglab/emalign.git"
+```
+
+PanPhon keeps only symbols it knows. For example, it drops the tone numbers of Middle Chinese protoforms (`kʰwen²` becomes `kʰwen`), but keeps tone letters such as `˥˩`. It also keeps sequences like `ts` as two sounds unless a tie bar joins them (`t͡s`).
+
+## Results
+
+Cognates and alignments have three parts, `(protoform, reflexes, auxiliary)`, and you only need the first two:
+
+```python
+Cognate(protoform='ka', reflexes={'4': 'ka', '1': 'ku', '2': 'ke', '3': 'ki'}, auxiliary={...})
+```
+
+`auxiliary` carries extra information from the dataset that the print functions use, such as the gloss, the cognate set's ID, the proto-language and the language names.
+
+## Data functions
+
+**0. `load_cldf(path)`** reads a CLDF dataset folder.
+
+**1. `cognates(data, proto=None, langs=None)`** gives one `Cognate` per cognate set:
+- `protoform` is the reconstruction in IPA, and `reflexes` holds each daughter language's word in IPA, e.g. `{'4': 'ka', '1': 'ku'}`.
+- `proto` keeps only the sets of that proto-language, and `langs` picks the daughter languages and their order. By default you get every set and every daughter language.
+- Each language gets one word per set (the first listed), and only the part of the word that is cognate. Parts are separated by `+` or spaces.
+
+**2. `align_cognates(cogs)`** lines up the protoform and the words sound by sound, using `-` for gaps. emalign learns how similar sounds are from all the sets at once, so the sets are aligned together. Columns where no word has a sound are removed. All the lists of one set have the same length, e.g. `protoform=['k', 'a'], reflexes={'4': ['k', 'a'], '1': ['k', 'u']}`.
+
+**3. `correspond(aligned, window=0)`** turns the alignments into rules (see below). Each distinct pattern of daughter sounds becomes one rule, concluding the proto sound it most often lines up with, and the most frequent rules come first. With `window=1`, the pattern also includes the sound before and after, which separates more cases.
+
+**4. `predict(rules, aligned)`** rebuilds each protoform from the daughters' aligned sounds, column by column. It returns a `Cognate` per set with the predicted protoform, to compare with `cogs[i].protoform`.
+
+## Rules
+
+Rules use the same format as `pi_logic.py`: `(conditions, conclusion)`, read as "if every condition holds, the proto sound is the conclusion". A condition maps a key, `Key(language ID, offset)`, to a sound. A `Key` is a named tuple equal to the plain tuple `("4", 0)`, so either can be written:
+
+```python
+rule = ({("4", 0): "r", ("1", 0): "ð"}, "ð")        # Ukhrul r and Kachai ð come from *ð
+rule = ({("4", 0): "k", ("4", -1): "^"}, "k")       # Ukhrul k at the start of a word
+```
+
+- `("4", 0)` is language 4's sound in the current column. `("4", -1)` is the sound before it, and `("4", 1)` the sound after.
+- `"^"` means before the start of the word, `"$"` after its end, and `"-"` a gap.
+- A rule applies when all its conditions hold. If several rules apply, the one with the most conditions wins, and ties go to the earlier rule.
+- A conclusion of `""` or `"-"` adds nothing, and so does a column that no rule matches.
+
+`correspond` returns `Rule(conditions, conclusion, auxiliary)`, whose condition keys are `Key`s, equal to the tuples of hand-written rules, so the two mix freely (e.g. `predict(my_rules + rules, aligned)`). Its `auxiliary` holds how often the rule's pattern occurs, how often each other proto sound occurs with the same pattern (these are the cases it can't separate), and the proto sounds before and after it if they are the same every time. Because rules are `(conditions, conclusion)` pairs, they work directly with `pi_logic`, e.g. `ImplicationSystem.from_tuples(rule[:2] for rule in rules)`.
+
+## Evaluation functions
+
+**`evaluate(rules, aligned, weight=2.0)`** scores a rule set the same way as `iterative_update_pi.py`: **L = 2E + wR**.
+- **E** is the number of cognate sets whose predicted protoform differs from the attested one.
+- **R** is the number of rules, and **w** is `weight`.
+
+It returns `Loss(total, errors, rules)`, e.g. `Loss(total=1242.0, errors=73, rules=548)`. A lower total is better, so compare two rule sets with `evaluate(a, aligned).total < evaluate(b, aligned).total`.
+
+**`metrics(predicted)`** measures how close the predictions from `predict` are to the attested protoforms. It returns `Metrics(accuracy, per, pfer)`, e.g. `Metrics(accuracy=0.94, per=0.021, pfer=0.010)`:
+- **accuracy** is the share of protoforms predicted exactly.
+- **per** (phoneme error rate) is the number of sounds to insert, delete or substitute to turn the predictions into the attested protoforms, per attested sound.
+- **pfer** (phonetic feature error rate) is the same, but a substitution costs only the share of PanPhon features that differ, so near misses like *t* for *d* count less.
+
+## Display functions
+
+Each takes the output of one data function:
+
+- **`print_cognates(cogs)`**: a row per set, with its protoform (in a column titled with the proto-language's name), gloss, and each language's word.
+- **`print_aligned(aligned[i])`**: one aligned set, with a row per language (name, ID) and a column per sound.
+- **`print_rules(rules)`**: a row per rule, written as a sound change, `*k → k ʔ ∅ / _#`. It reads "proto *k becomes k, ʔ and nothing at the end of a word", and the count is at the end. A condition at an offset gets a column like `Ukhrul -1`.
+- **`print_predicted(predicted)`**: a row per set, with its gloss, the attested protoform (under the proto-language's name) and the predicted one.
+- **`print_loss(loss)`**: the loss and its parts (errors, rules, weight). Pass a list of losses to compare several rule sets, one row each.
+- **`print_metrics(scores)`**: accuracy, PER and PFER, also for one `Metrics` or a list of them.
+
+In the tables, the first row has the language names and the second row their IDs. The data starts on the third row, numbered from 0, so row `i` is `cogs[i]`. A list that mixes proto-languages gets the title "Protoform" instead of a name. A table wider than the terminal is split into blocks that fit, each repeating the first columns (row number, protoform, gloss), so every language stays readable however many there are. To show only some languages, pick them in `cognates(data, langs=[...])`.
+
+## Running
+
+From `cldf-utils`, run `python3 -i with_datasets.py` to open Python with the four datasets loaded as `au` (Austronesian), `ur` (Uralic), `ch` (Chinese) and `ta` (Tangkhulic). The file's comments list commands to try.

acore/cldf_utils.py

@@ -0,0 +1,345 @@
+"""Comparative-method utilities for CLDF comparative dictionaries (guide: cldf_utils.md)."""
+import io, re, shutil, unicodedata
+from collections import Counter, defaultdict, namedtuple
+from functools import lru_cache
+from pathlib import Path
+import pandas as pd
+import emalign.cldf_io
+import panphon.distance
+
+from acore.ipa import GAP, START, END, BOUNDARY, NULL, segment as _segment
+
+ARROW = "→"
+
+Cognate = namedtuple("Cognate", "protoform reflexes auxiliary", defaults=[{}])
+Alignment = namedtuple("Alignment", "protoform reflexes auxiliary", defaults=[{}])
+Rule = namedtuple("Rule", "conditions conclusion auxiliary", defaults=[{}])
+Key = namedtuple("Key", "lang offset", defaults=[0])
+Loss = namedtuple("Loss", "total errors rules")
+Metrics = namedtuple("Metrics", "accuracy per pfer")
+
+
+# Helpers for the data functions ---------------------------------------------------------------
+
+
+def _protoform(name: str) -> str:
+    return "".join(_segment(re.split(r"[(,~/]", name, maxsplit=1)[0]))
+
+
+def _morph(form: str, index: str) -> str:
+    morphs, i = [m for m in re.split(r"[+ ]", form) if m], int(index) if index.isdigit() else 0
+    return morphs[i] if i < len(morphs) else ""
+
+
+def _join(segs) -> str:
+    return "".join(seg for seg in segs if seg != GAP)
+
+
+def _gapless(rows) -> list[int]:
+    return [i for i, column in enumerate(zip(*rows)) if set(column) != {GAP}]
+
+
+def _daughters(a) -> tuple[list[int], dict[str, list[str]]]:
+    keep = _gapless(a.reflexes.values())
+    return keep, {lang: [row[i] for i in keep] for lang, row in a.reflexes.items()}
+
+
+def _context(rows: dict, i: int, window: int) -> dict[Key, str]:
+    return {Key(lang, offset): row[i + offset] if 0 <= i + offset < len(row)
+            else START if i + offset < 0 else END
+            for lang, row in rows.items() for offset in range(-window, window + 1)}
+
+
+def _conditions(conditions) -> dict[Key, str]:
+    return {Key(str(lang), int(offset)): unicodedata.normalize("NFC", seg)
+            for (lang, offset), seg in conditions.items()}
+
+
+# Data functions: CLDF folder -> cognates -> alignments -> rules -> predictions -----------------
+
+def load_cldf(path: str | Path) -> dict[str, pd.DataFrame]:
+    """Read a CLDF folder into one DataFrame per table (languages, forms, cognates, ...)."""
+    folder = Path(path) / "cldf" if (Path(path) / "cldf").is_dir() else Path(path)
+    data = {}
+    for name in ("languages", "forms", "cognates", "cognatesets", "etyma", "parameters"):
+        if (file := folder / f"{name}.csv").exists():
+            text = file.read_bytes().decode("utf-8-sig").replace("\r", "")
+            data[name] = pd.read_csv(io.StringIO(text), dtype=str, keep_default_na=False)
+    return data
+
+
+def cognates(data, proto=None, langs=None) -> list[Cognate]:
+    """Cognate sets of proto-language ``proto`` (default: all), with the words of ``langs``
+    (default: every daughter language), in that order. Languages are given by CLDF ID."""
+    languages = data["languages"]
+    if langs is None:
+        langs = languages.ID[languages.Is_Proto.str.lower() != "true"]
+    rank = {str(lang): r for r, lang in enumerate(langs)}
+    etyma = data["etyma"].set_index("ID")
+    sets = data["cognatesets"].set_index("ID").Etymon_ID
+    sets = sets[sets.isin(etyma.index)]
+    protos = sets.map(etyma.Language_ID)
+    if proto is not None:
+        sets = sets[protos == str(proto)]
+    protoforms = sets.map(etyma.Name).map(_protoform)
+    forms = data["forms"][data["forms"].Language_ID.isin(rank)]
+    links = data["cognates"][data["cognates"].Cognateset_ID.isin(sets.index)].merge(
+        forms[["ID", "Language_ID", "Local_ID", "Form"]], left_on="Form_ID", right_on="ID")
+    links = links.assign(word=[_morph(f, i) for f, i in zip(links.Form, links.Morph_Index)],
+                         rank=links.Language_ID.map(rank))
+    links = links[links.word.map(_segment).astype(bool)].sort_values(
+        "Local_ID", key=lambda ids: pd.to_numeric(ids, errors="coerce"), kind="stable")
+    links = links.drop_duplicates(["Cognateset_ID", "Language_ID"])
+    links = links.sort_values("rank", kind="stable")
+    reflexes = defaultdict(dict)
+    for set_id, lang, word in zip(links.Cognateset_ID, links.Language_ID, links.word):
+        reflexes[set_id][lang] = word
+    names = dict(zip(languages.ID, languages.Name))
+    glosses = dict(zip(data["cognatesets"].ID, data["cognatesets"].Description))
+    protos, cogs = protos.to_dict(), []
+    for set_id, form in protoforms.items():
+        if form and set_id in reflexes:
+            ids = (protos[set_id], *reflexes[set_id])
+            aux = {"id": set_id, "proto": protos[set_id], "gloss": glosses.get(set_id, ""),
+                   "names": {lang: names.get(lang, lang) for lang in ids}}
+            cogs.append(Cognate(form, reflexes[set_id], aux))
+    return cogs
+
+
+def align_cognates(cogs: list[Cognate]) -> list[Alignment]:
+    """Align each set's words and protoform segment by segment ('-' = gap) with emalign, fitted
+    on all the sets, so that all the lists of one set have the same length."""
+    sets, rows = [], {}
+    for n, cog in enumerate(cogs):
+        words = [" ".join(_segment(word)) for word in (*cog.reflexes.values(), cog.protoform)]
+        forms = [(f"{n}.{k}", word) for k, word in enumerate(words)]
+        rows.update((i, word.split()) for i, word in forms)
+        entries = [(emalign.cldf_io.CognateEntry(i, i, str(n), 0),
+                    emalign.cldf_io.Form(i, i, word, [word])) for i, word in forms]
+        sets.append(emalign.cldf_io.CognateSet(str(n), entries))
+    rows.update((r.form_id, unicodedata.normalize("NFC", r.aligned_form).split("|"))
+                for r in emalign.CognateAligner(random_seed=0).fit_and_align(sets))
+    aligned = []
+    for n, cog in enumerate(cogs):
+        table = [rows[f"{n}.{k}"] for k in range(len(cog.reflexes) + 1)]
+        keep = _gapless(table)
+        table = [[row[i] for i in keep] for row in table]
+        aligned.append(Alignment(table[-1], dict(zip(cog.reflexes, table)), cog.auxiliary))
+    return aligned
+
+
+def correspond(aligned: list[Alignment], window: int = 0) -> list[Rule]:
+    """One rule per distinct context (the daughters' segments within ``window`` columns of a
+    column), concluding the proto segment most often aligned with it; most frequent first. The
+    auxiliary data holds its count, every conclusion's count, and the proto environment
+    (before, after) shared by all its occurrences ('' where they differ)."""
+    counts, first, envs = defaultdict(Counter), {}, defaultdict(lambda: (set(), set()))
+    for a in aligned:
+        keep, rows = _daughters(a)
+        sounds = [i for i, seg in enumerate(a.protoform) if seg != GAP]
+        for n, i in enumerate(keep):
+            context, proto = tuple(_context(rows, n, window).items()), a.protoform[i]
+            counts[context][proto] += 1
+            first.setdefault(context, a.auxiliary)
+            before, after = envs[context, proto]
+            before.add(next((a.protoform[j] for j in reversed(sounds) if j < i), BOUNDARY))
+            after.add(next((a.protoform[j] for j in sounds if j > i), BOUNDARY))
+    rules = []
+    for context, protos in counts.items():
+        proto, count = protos.most_common(1)[0]
+        env = tuple(next(iter(side)) if len(side) == 1 else "" for side in envs[context, proto])
+        aux = {"count": count, "protos": dict(protos.most_common()), "env": env,
+               "proto": first[context].get("proto", ""), "names": first[context].get("names", {})}
+        rules.append(Rule(dict(context), proto, aux))
+    return sorted(rules, key=lambda rule: -rule.auxiliary["count"])
+
+
+def predict(rules, aligned: list[Alignment]) -> list[Cognate]:
+    """Predict each set's protoform from its daughters' aligned segments. In each column, the
+    matching rule with the most conditions (the earlier one on ties) adds its conclusion."""
+    rules = [(_conditions(rule[0]), unicodedata.normalize("NFC", rule[1])) for rule in rules]
+    rules.sort(key=lambda r: -len(r[0]))
+    reach = max((abs(key.offset) for conds, _ in rules for key in conds), default=0)
+    frequency = Counter(item for conds, _ in rules for item in conds.items())
+    index = defaultdict(list)
+    for n, (conds, _) in enumerate(rules):
+        index[min(conds.items(), key=frequency.__getitem__, default=None)].append(n)
+    predicted = []
+    for a in aligned:
+        keep, rows = _daughters(a)
+        proto = []
+        for i in range(len(keep)):
+            context = _context(rows, i, reach)
+            hits = {*index[None], *(n for item in context.items() for n in index.get(item, ()))}
+            matches = (rules[n][1] for n in sorted(hits)
+                       if all(context.get(key) == seg for key, seg in rules[n][0].items()))
+            proto.append(next(matches, ""))
+        words = {lang: _join(row) for lang, row in rows.items()}
+        aux = {**a.auxiliary, "reference": _join(a.protoform)}
+        predicted.append(Cognate(_join(proto), words, aux))
+    return predicted
+
+
+# Evaluation functions: the loss of iterative_update_pi, and accuracy and error rates -----------
+
+def evaluate(rules, aligned: list[Alignment], weight: float = 2.0) -> Loss:
+    """Loss of ``rules`` on ``aligned``: L = 2E + wR, where E counts the sets whose predicted
+    protoform differs from the attested one, R counts the rules and w is ``weight``."""
+    rules = list(rules)
+    errors = sum(cog.protoform != cog.auxiliary["reference"] for cog in predict(rules, aligned))
+    return Loss(2 * errors + weight * len(rules), errors, len(rules))
+
+
+@lru_cache(maxsize=None)
+def _distance():
+    return panphon.distance.Distance()
+
+
+def metrics(predicted: list[Cognate]) -> Metrics:
+    """Accuracy (share of protoforms predicted exactly), phoneme error rate (segment edit
+    distance) and PanPhon feature error rate (feature edit distance), both per attested segment."""
+    pairs = [(cog.protoform, cog.auxiliary["reference"]) for cog in predicted]
+    d, one, differ = _distance(), lambda _: 1, lambda a, b: a != b
+    edits = sum(d.min_edit_distance(one, one, differ, [""], list(_segment(p)), list(_segment(r)))
+                for p, r in pairs)
+    features = sum(d.feature_edit_distance(p, r) for p, r in pairs)
+    segments = max(sum(len(_segment(r)) for _, r in pairs), 1)
+    exact = sum(p == r for p, r in pairs) / max(len(pairs), 1)
+    return Metrics(exact, edits / segments, features / segments)
+
+
+# Display functions: print the data functions' outputs as plain, aligned tables ------------------
+
+def _aux(item) -> dict:
+    return item[2] if len(item) > 2 else {}
+
+
+def _names(items) -> dict[str, str]:
+    return {lang: name for item in items for lang, name in _aux(item).get("names", {}).items()}
+
+
+def _proto_name(aux: dict) -> str:
+    return aux.get("names", {}).get(aux.get("proto"), aux.get("proto", ""))
+
+
+def _star(form: str) -> str:
+    return f"*{form}" if form else ""
+
+
+def _null(seg: str) -> str:
+    return NULL if seg in (GAP, "") else seg
+
+
+def _clip(text: str, limit: int = 24) -> str:
+    return text if len(text) <= limit else text[:limit - 1].rstrip() + "…"
+
+
+def _width(text: str) -> int:
+    return sum(0 if unicodedata.category(char) in ("Mn", "Me", "Cf")
+               else 2 if unicodedata.east_asian_width(char) in ("W", "F") else 1 for char in text)
+
+
+def _order(dicts) -> list:
+    order = []
+    for keys in map(list, dicts):
+        for i, key in enumerate(keys):
+            if key not in order:
+                later = [order.index(k) for k in keys[i + 1:] if k in order]
+                order.insert(min(later, default=len(order)), key)
+    return order
+
+
+def _proto(items, default: str) -> tuple[str, str]:
+    protos = {(_proto_name(aux), aux["proto"]) for aux in map(_aux, items) if aux.get("proto")}
+    return protos.pop() if len(protos) == 1 else (default, "")
+
+
+def _header(*cells: tuple[str, str]) -> list[list[str]]:
+    return [[title for title, _ in cells], [code for _, code in cells]]
+
+
+def _print_table(rows: list[list[str]], frozen: int = 1) -> None:
+    widths = [max(map(_width, column)) for column in zip(*rows)]
+    room = shutil.get_terminal_size().columns - sum(widths[:frozen]) - 2 * frozen
+    used, blocks = 0, [[]]
+    for c in range(frozen, len(widths)):
+        if blocks[-1] and used + widths[c] > room:
+            blocks.append([])
+            used = 0
+        blocks[-1].append(c)
+        used += widths[c] + 2
+    for n, block in enumerate(blocks):
+        if n:
+            print()
+        for row in rows:
+            print("  ".join(row[c] + " " * (widths[c] - _width(row[c]))
+                            for c in (*range(frozen), *block)).rstrip())
+
+
+def print_cognates(cogs: list[Cognate]) -> None:
+    """Print cognate sets as a table: protoform (titled by its proto-language), gloss, then each
+    language's word. The second row holds the language IDs."""
+    langs, names = _order(cog.reflexes for cog in cogs), _names(cogs)
+    rows = _header(("", ""), _proto(cogs, "Protoform"), ("Gloss", ""),
+                   *((names.get(lang, lang), lang) for lang in langs))
+    for n, cog in enumerate(cogs):
+        rows.append([str(n), _star(cog.protoform), _clip(_aux(cog).get("gloss", "")),
+                     *(cog.reflexes.get(lang, "") for lang in langs)])
+    _print_table(rows, frozen=3)
+
+
+def print_aligned(a: Alignment) -> None:
+    """Print one aligned cognate set: a row per language with its name and ID, and a column per
+    segment."""
+    aux = _aux(a)
+    rows = [["", "", *map(str, range(len(a.protoform)))],
+            [_proto_name(aux) or "Protoform", aux.get("proto", ""), *a.protoform]]
+    rows += [[aux.get("names", {}).get(lang, lang), lang, *row] for lang, row in a.reflexes.items()]
+    print(aux.get("gloss", ""))
+    _print_table(rows, frozen=2)
+
+
+def print_rules(rules: list[Rule]) -> None:
+    """Print rules as sound changes, *conclusion → each condition's segment / environment, with
+    how often each occurs. A condition with an offset is titled e.g. Ukhrul -1."""
+    conds = [_conditions(rule[0]) for rule in rules]
+    keys, names = _order(conds), _names(rules)
+    rows = _header(("", ""), _proto(rules, "Protoform"), ("", ""),
+                   *((names.get(lang, lang) + (f" {offset:+d}" if offset else ""), lang)
+                     for lang, offset in keys), ("", ""), ("Count", ""))
+    for n, (rule, cond) in enumerate(zip(rules, conds)):
+        aux = _aux(rule)
+        before, after = aux.get("env", ("", ""))
+        segs = (_null(cond[key]) if key in cond else "" for key in keys)
+        rows.append([str(n), NULL if rule[1] in (GAP, "") else _star(rule[1]), ARROW, *segs,
+                     f"/ {before}_{after}" if before or after else "", str(aux.get("count", ""))])
+    _print_table(rows, frozen=3)
+
+
+def print_predicted(predicted: list[Cognate]) -> None:
+    """Print each set's gloss, attested protoform (titled by its proto-language) and predicted
+    protoform. The second row holds the proto-language ID."""
+    rows = _header(("", ""), ("Gloss", ""), _proto(predicted, "Reference"), ("Predicted", ""))
+    for n, cog in enumerate(predicted):
+        aux = _aux(cog)
+        rows.append([str(n), _clip(aux.get("gloss", "")), _star(aux.get("reference", "")),
+                     _star(cog.protoform)])
+    _print_table(rows)
+
+
+def print_loss(loss) -> None:
+    """Print a Loss, or a list of them (e.g. one per rule set), as L = 2E + wR and its parts."""
+    rows = [["", "Loss", "Errors", "Rules", "Weight"]]
+    for n, (total, errors, rules) in enumerate([loss] if isinstance(loss, Loss) else loss):
+        weight = f"{(total - 2 * errors) / rules:.10g}" if rules else ""
+        rows.append([str(n), f"{total:.10g}", str(errors), str(rules), weight])
+    print("L = 2E + wR")
+    _print_table(rows)
+
+
+def print_metrics(scores) -> None:
+    """Print a Metrics, or a list of them (e.g. one per rule set): accuracy, PER and PFER."""
+    rows = [["", "Accuracy", "PER", "PFER"]]
+    rows += [[str(n), *(f"{value:.3f}" for value in values)]
+             for n, values in enumerate([scores] if isinstance(scores, Metrics) else scores)]
+    _print_table(rows)

acore/cldf_utils.pyi

@@ -0,0 +1,71 @@
+# Interface of cldf_utils.py. Guide: cldf_utils.md.
+# Languages are identified by their CLDF ID, the ID column of languages.csv ('19094', 'Finnish').
+# IPA is segmented with PanPhon's FeatureTable.ipa_segs; alignment uses emalign.
+
+from collections.abc import Iterable, Mapping
+from pathlib import Path
+from typing import Any, NamedTuple
+
+import pandas as pd
+
+# Cognates and alignments are (protoform, reflexes, auxiliary); rules are (conditions,
+# conclusion, auxiliary), the pi_logic format. Work with the first two fields; auxiliary carries
+# extra CLDF information for the display functions (default {}):
+#   Cognate, Alignment: {'id': cognate set ID, 'proto': proto-language ID, 'gloss': str,
+#                        'names': {language ID: name}}, plus 'reference' from predict
+#   Rule: {'count': int, 'protos': {conclusion: count}, 'env': (before, after), 'proto', 'names'}
+Aux = dict[str, Any]
+Cognate = NamedTuple("Cognate", [("protoform", str), ("reflexes", dict[str, str]),
+                                 ("auxiliary", Aux)])
+Alignment = NamedTuple("Alignment", [("protoform", list[str]),
+                                     ("reflexes", dict[str, list[str]]), ("auxiliary", Aux)])
+
+# A condition key is Key(language ID, offset), equal to the tuple ('4', 0): that language's segment
+# at an offset from the current column. Past the edges of a word it is '^' (start) or '$' (end).
+Key = NamedTuple("Key", [("lang", str), ("offset", int)])
+Rule = NamedTuple("Rule", [("conditions", dict[Key, str]), ("conclusion", str),
+                           ("auxiliary", Aux)])
+
+# Any (conditions, conclusion) pair is a rule, e.g. ({('4', 0): 'r', ('1', -1): '^'}, 'ð').
+Rules = Iterable[Rule | tuple[Mapping[tuple[str, int], str], str]]
+
+# Data functions
+
+def load_cldf(path: str | Path) -> dict[str, pd.DataFrame]: ...
+
+def cognates(
+    data: dict[str, pd.DataFrame], proto: str | None = ..., langs: Iterable[str] | None = ...
+) -> list[Cognate]: ...
+
+def align_cognates(cogs: list[Cognate]) -> list[Alignment]: ...  # equal-length lists per set
+
+def correspond(aligned: list[Alignment], window: int = ...) -> list[Rule]: ...  # most frequent
+
+def predict(rules: Rules, aligned: list[Alignment]) -> list[Cognate]: ...
+
+# Evaluation functions
+
+# L = 2E + wR: E = sets whose predicted protoform differs from the attested one, R = rules.
+Loss = NamedTuple("Loss", [("total", float), ("errors", int), ("rules", int)])
+
+def evaluate(rules: Rules, aligned: list[Alignment], weight: float = ...) -> Loss: ...
+
+# accuracy: share of exact protoforms; per, pfer: segment and PanPhon feature edit distance per
+# attested segment.
+Metrics = NamedTuple("Metrics", [("accuracy", float), ("per", float), ("pfer", float)])
+
+def metrics(predicted: list[Cognate]) -> Metrics: ...
+
+# Display functions
+
+def print_cognates(cogs: list[Cognate]) -> None: ...
+
+def print_aligned(a: Alignment) -> None: ...  # one set
+
+def print_rules(rules: Rules) -> None: ...
+
+def print_predicted(predicted: list[Cognate]) -> None: ...
+
+def print_loss(loss: Loss | list[Loss]) -> None: ...
+
+def print_metrics(scores: Metrics | list[Metrics]) -> None: ...

acore/ipa.py

@@ -0,0 +1,112 @@
+"""IPA segmentation and normalization utilities using panphon.
+
+This module provides consistent IPA tokenization across the acore library
+using panphon's FeatureTable.ipa_segs() method.
+"""
+
+from __future__ import annotations
+
+import unicodedata
+from functools import lru_cache
+from typing import TYPE_CHECKING
+
+import panphon
+
+if TYPE_CHECKING:
+    from collections.abc import Sequence
+
+# Shared FeatureTable instance
+_FT = panphon.FeatureTable()
+
+# Common symbols
+GAP = "-"
+START = "^"
+END = "$"
+BOUNDARY = "#"
+NULL = "∅"
+
+
+@lru_cache(maxsize=None)
+def segment(form: str, normalize: bool = True) -> tuple[str, ...]:
+    """Segment an IPA string into phonemes using panphon.
+
+    Args:
+        form: IPA string to segment.
+        normalize: If True, normalize each segment to NFC.
+
+    Returns:
+        Tuple of IPA segments (phonemes).
+
+    Example:
+        >>> segment("kʰat")
+        ('kʰ', 'a', 't')
+    """
+    # Replace ASCII 'g' with IPA 'ɡ' (U+0261)
+    form = form.replace("g", "ɡ")
+    segments = _FT.ipa_segs(form)
+    if normalize:
+        return tuple(unicodedata.normalize("NFC", seg) for seg in segments)
+    return tuple(segments)
+
+
+def join(segments: Sequence[str], exclude_gaps: bool = True) -> str:
+    """Join segments into a string.
+
+    Args:
+        segments: Sequence of IPA segments.
+        exclude_gaps: If True, exclude GAP symbols from output.
+
+    Returns:
+        Joined string.
+    """
+    if exclude_gaps:
+        return "".join(seg for seg in segments if seg != GAP)
+    return "".join(segments)
+
+
+def normalize(form: str, form_type: str = "NFC") -> str:
+    """Normalize a Unicode string.
+
+    Args:
+        form: String to normalize.
+        form_type: Unicode normalization form (NFC, NFD, NFKC, NFKD).
+
+    Returns:
+        Normalized string.
+    """
+    return unicodedata.normalize(form_type, form)
+
+
+def segment_to_string(form: str) -> str:
+    """Segment an IPA string and rejoin it.
+
+    This ensures proper segmentation boundaries are respected.
+
+    Args:
+        form: IPA string.
+
+    Returns:
+        Re-joined string after segmentation.
+    """
+    return join(segment(form), exclude_gaps=False)
+
+
+def is_ipa(char: str) -> bool:
+    """Check if a character is a valid IPA character.
+
+    Args:
+        char: Single character to check.
+
+    Returns:
+        True if the character is recognized by panphon.
+    """
+    return _FT.fts(char) is not None
+
+
+def feature_table() -> panphon.FeatureTable:
+    """Return the shared panphon FeatureTable instance.
+
+    Returns:
+        The panphon FeatureTable used for IPA operations.
+    """
+    return _FT

acore/pi/__init__.py

@@ -0,0 +1,34 @@
+"""Pi submodule for proto-language reconstruction using implication systems.
+
+This module provides tools for:
+- Applying pi (implication systems) to cognate sets to generate protoforms
+- Initializing pi from cognate sets and known protoforms
+"""
+
+from acore.pi.apply import (
+    Condition,
+    Implication,
+    apply_pi,
+    apply_pi_to_alignment,
+    load_pi,
+    reconstruct_from_cldf,
+)
+from acore.pi.initialize import (
+    OptimizationResult,
+    initialize_pi,
+    initialize_pi_from_cldf,
+    save_pi,
+)
+
+__all__ = [
+    "apply_pi",
+    "apply_pi_to_alignment",
+    "Condition",
+    "Implication",
+    "initialize_pi",
+    "initialize_pi_from_cldf",
+    "load_pi",
+    "OptimizationResult",
+    "reconstruct_from_cldf",
+    "save_pi",
+]