retexo 0.1.0__tar.gz

retexo-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Julian Schelb, Michael Wittweiler, Marie Revellio
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

retexo-0.1.0/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.4
+Name: retexo
+Version: 0.1.0
+Summary: Retexo explains Latin text reuse word by word: it aligns a reusing passage to its source and names the operation behind every link.
+License: MIT
+License-File: LICENSE
+Keywords: latin,intertextuality,text reuse,word alignment,edit scripts,digital humanities
+Author: Julian Schelb
+Author-email: julian.schelb@uni-konstanz.de
+Requires-Python: >=3.10,<3.15
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: Latin
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Provides-Extra: dev
+Provides-Extra: gui
+Provides-Extra: lexical
+Provides-Extra: llm
+Provides-Extra: plots
+Requires-Dist: anthropic (>=0.30.0) ; extra == "llm"
+Requires-Dist: build ; extra == "dev"
+Requires-Dist: cltk (>=1.1.0,<2.0.0) ; (python_version < "3.13") and (extra == "lexical")
+Requires-Dist: gensim (>=4.3.0,<5.0.0) ; extra == "lexical"
+Requires-Dist: gradio (>=5.49.1) ; extra == "gui"
+Requires-Dist: huggingface-hub (>=0.20.0)
+Requires-Dist: locisimiles (>=2.1.3)
+Requires-Dist: matplotlib (>=3.7.0) ; extra == "plots"
+Requires-Dist: mkdocs (>=1.5.0) ; extra == "dev"
+Requires-Dist: mkdocs-material (>=9.0.0) ; extra == "dev"
+Requires-Dist: mkdocstrings[python] (>=0.24.0) ; extra == "dev"
+Requires-Dist: nltk (>=3.8.0) ; extra == "lexical"
+Requires-Dist: numpy (>=1.24.0,<3.0.0)
+Requires-Dist: pandas (>=2.0.0,<3.0.0)
+Requires-Dist: peft (>=0.10.0) ; extra == "llm"
+Requires-Dist: poethepoet (>=0.24.0) ; extra == "dev"
+Requires-Dist: pre-commit (>=3.5.0) ; extra == "dev"
+Requires-Dist: pytest (>=8.0.0,<10.0.0) ; extra == "dev"
+Requires-Dist: pytest-cov (>=4.0.0) ; extra == "dev"
+Requires-Dist: ruff (>=0.8.0) ; extra == "dev"
+Requires-Dist: scikit-learn (>=1.3.0,<2.0.0)
+Requires-Dist: scipy (>=1.10.0,<2.0.0)
+Requires-Dist: spacy (>=3.5.0,<4.0.0) ; extra == "lexical"
+Requires-Dist: stanza (>=1.7.0) ; extra == "lexical"
+Requires-Dist: torch (>=2.0.0,<3.0.0)
+Requires-Dist: transformers (>=4.30.0,<5.0.0)
+Requires-Dist: twine ; extra == "dev"
+Project-URL: Changelog, https://github.com/julianschelb/retexo/blob/main/CHANGELOG.md
+Project-URL: Documentation, https://julianschelb.github.io/retexo/
+Project-URL: Demo, https://julianschelb.github.io/retexo/demo/
+Project-URL: Dataset, https://huggingface.co/datasets/julian-schelb/latin-classical-intertextuality-edit-scripts
+Project-URL: Homepage, https://julianschelb.github.io/retexo/
+Project-URL: Issues, https://github.com/julianschelb/retexo/issues
+Project-URL: Repository, https://github.com/julianschelb/retexo
+Description-Content-Type: text/markdown
+
+# Retexo
+
+[![CI](https://github.com/julianschelb/retexo/actions/workflows/ci.yml/badge.svg)](https://github.com/julianschelb/retexo/actions/workflows/ci.yml)
+[![Docs](https://img.shields.io/badge/docs-julianschelb.github.io%2Fretexo-blue)](https://julianschelb.github.io/retexo/)
+[![Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20dataset-edit--scripts-yellow)](https://huggingface.co/datasets/julian-schelb/latin-classical-intertextuality-edit-scripts)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/julianschelb/retexo/blob/main/LICENSE)
+
+**Word-level explanations of text reuse in Latin literature.**
+
+When a later author quotes or alludes to an earlier one, the interesting question is *how* the source was reworked.
+Retexo answers it word by word: it aligns every word of the reusing passage to the source word it takes up, or to none,
+and names the operation behind each link, such as a copy, a change of inflection, or a substitution. The result is an
+*edit script*: a set of labeled links between two passages that a philologist can read and contest.
+
+Retexo learns these scripts from very little annotation. Synthetic pairs, built by applying known operations to Latin
+text, teach a pointer network the task before a single pair is annotated, and active learning chooses the few pairs an
+annotator corrects.
+
+- **Predicted edit scripts** for all 1,490 references of the [Loci Similes](https://arxiv.org/abs/2601.07533) benchmark:
+  [Hugging Face dataset](https://huggingface.co/datasets/julian-schelb/latin-classical-intertextuality-edit-scripts)
+- **Review app** to browse the script of every reference, drawn as arrows between the passages:
+  [julianschelb.github.io/retexo/demo](https://julianschelb.github.io/retexo/demo/)
+- **Documentation:** [julianschelb.github.io/retexo](https://julianschelb.github.io/retexo/)
+
+## Quick start
+
+```bash
+pip install "retexo @ git+https://github.com/julianschelb/retexo"
+```
+
+Read the predictions, with the Hugging Face `datasets` library:
+
+```python
+from datasets import load_dataset
+
+scripts = load_dataset("julian-schelb/latin-classical-intertextuality-edit-scripts", split="fold_4")
+pair = scripts[0]
+
+for link in pair["links"]:
+    print(pair["reuse"]["tokens"][link["reuse"]], "<-", pair["source"]["tokens"][link["source"]], link["label"])
+```
+
+Convert the raw predictions of a run into the same format:
+
+```bash
+retexo export runs/f0/predictions.jsonl runs/f1/predictions.jsonl runs/f2/predictions.jsonl \
+              runs/f3/predictions.jsonl runs/f4/predictions.jsonl --out export/ --format parquet
+```
+
+## Labels
+
+| Label | Meaning |
+|---|---|
+| `COPY` | the same form, spelling variants folded |
+| `INFLECT` | the same lemma in another form |
+| `SUBST` | another lemma in the same slot, optionally with a named relation such as a synonym |
+| `SPLIT`, `MERGE` | two reuse words for one source word, or one for two |
+| `INS`, `FRAME`, `DEL` | an unlinked reuse word, a word of a citing formula, an unclaimed source word |
+
+See the [documentation](https://julianschelb.github.io/retexo/labels/) for the conventions.
+
+## Repository layout
+
+```
+src/retexo/         the package: model, generator, active learning, baselines, scorer
+src/retexo_gui/     Gradio demo and annotation app (gui extra)
+tests/              pytest suite
+examples/           notebooks on the operations, the generator, the passage class and the decoder
+docs/               documentation (MkDocs)
+webapp/             the review app (React, Vite), published under /demo/
+```
+
+The experiments behind the paper live in a separate repository that installs retexo as a dependency.
+
+## Development
+
+```bash
+pip install -e ".[dev,lexical]"
+pytest
+mkdocs serve
+```
+
+See [docs/development.md](https://julianschelb.github.io/retexo/development/).
+
+## Authors
+
+Julian Schelb (University of Konstanz), Michael Wittweiler (University of Zurich), Marie Revellio (University of Konstanz).
+
+## License
+
+MIT
+

retexo-0.1.0/README.md

@@ -0,0 +1,91 @@
+# Retexo
+
+[![CI](https://github.com/julianschelb/retexo/actions/workflows/ci.yml/badge.svg)](https://github.com/julianschelb/retexo/actions/workflows/ci.yml)
+[![Docs](https://img.shields.io/badge/docs-julianschelb.github.io%2Fretexo-blue)](https://julianschelb.github.io/retexo/)
+[![Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20dataset-edit--scripts-yellow)](https://huggingface.co/datasets/julian-schelb/latin-classical-intertextuality-edit-scripts)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/julianschelb/retexo/blob/main/LICENSE)
+
+**Word-level explanations of text reuse in Latin literature.**
+
+When a later author quotes or alludes to an earlier one, the interesting question is *how* the source was reworked.
+Retexo answers it word by word: it aligns every word of the reusing passage to the source word it takes up, or to none,
+and names the operation behind each link, such as a copy, a change of inflection, or a substitution. The result is an
+*edit script*: a set of labeled links between two passages that a philologist can read and contest.
+
+Retexo learns these scripts from very little annotation. Synthetic pairs, built by applying known operations to Latin
+text, teach a pointer network the task before a single pair is annotated, and active learning chooses the few pairs an
+annotator corrects.
+
+- **Predicted edit scripts** for all 1,490 references of the [Loci Similes](https://arxiv.org/abs/2601.07533) benchmark:
+  [Hugging Face dataset](https://huggingface.co/datasets/julian-schelb/latin-classical-intertextuality-edit-scripts)
+- **Review app** to browse the script of every reference, drawn as arrows between the passages:
+  [julianschelb.github.io/retexo/demo](https://julianschelb.github.io/retexo/demo/)
+- **Documentation:** [julianschelb.github.io/retexo](https://julianschelb.github.io/retexo/)
+
+## Quick start
+
+```bash
+pip install "retexo @ git+https://github.com/julianschelb/retexo"
+```
+
+Read the predictions, with the Hugging Face `datasets` library:
+
+```python
+from datasets import load_dataset
+
+scripts = load_dataset("julian-schelb/latin-classical-intertextuality-edit-scripts", split="fold_4")
+pair = scripts[0]
+
+for link in pair["links"]:
+    print(pair["reuse"]["tokens"][link["reuse"]], "<-", pair["source"]["tokens"][link["source"]], link["label"])
+```
+
+Convert the raw predictions of a run into the same format:
+
+```bash
+retexo export runs/f0/predictions.jsonl runs/f1/predictions.jsonl runs/f2/predictions.jsonl \
+              runs/f3/predictions.jsonl runs/f4/predictions.jsonl --out export/ --format parquet
+```
+
+## Labels
+
+| Label | Meaning |
+|---|---|
+| `COPY` | the same form, spelling variants folded |
+| `INFLECT` | the same lemma in another form |
+| `SUBST` | another lemma in the same slot, optionally with a named relation such as a synonym |
+| `SPLIT`, `MERGE` | two reuse words for one source word, or one for two |
+| `INS`, `FRAME`, `DEL` | an unlinked reuse word, a word of a citing formula, an unclaimed source word |
+
+See the [documentation](https://julianschelb.github.io/retexo/labels/) for the conventions.
+
+## Repository layout
+
+```
+src/retexo/         the package: model, generator, active learning, baselines, scorer
+src/retexo_gui/     Gradio demo and annotation app (gui extra)
+tests/              pytest suite
+examples/           notebooks on the operations, the generator, the passage class and the decoder
+docs/               documentation (MkDocs)
+webapp/             the review app (React, Vite), published under /demo/
+```
+
+The experiments behind the paper live in a separate repository that installs retexo as a dependency.
+
+## Development
+
+```bash
+pip install -e ".[dev,lexical]"
+pytest
+mkdocs serve
+```
+
+See [docs/development.md](https://julianschelb.github.io/retexo/development/).
+
+## Authors
+
+Julian Schelb (University of Konstanz), Michael Wittweiler (University of Zurich), Marie Revellio (University of Konstanz).
+
+## License
+
+MIT

retexo-0.1.0/pyproject.toml

@@ -0,0 +1,118 @@
+[project]
+name = "retexo"
+version = "0.1.0"
+description = "Retexo explains Latin text reuse word by word: it aligns a reusing passage to its source and names the operation behind every link."
+authors = [
+    {name = "Julian Schelb", email = "julian.schelb@uni-konstanz.de"},
+    {name = "Michael Wittweiler", email = "michael.wittweiler@sglp.uzh.ch"},
+    {name = "Marie Revellio"},
+]
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.10,<3.15"
+keywords = ["latin", "intertextuality", "text reuse", "word alignment", "edit scripts", "digital humanities"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Natural Language :: Latin",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Text Processing :: Linguistic",
+]
+dependencies = [
+    "numpy (>=1.24.0,<3.0.0)",
+    "scipy (>=1.10.0,<2.0.0)",
+    "pandas (>=2.0.0,<3.0.0)",
+    "scikit-learn (>=1.3.0,<2.0.0)",
+    "torch (>=2.0.0,<3.0.0)",
+    "transformers (>=4.30.0,<5.0.0)",
+    "huggingface-hub (>=0.20.0)",
+    "locisimiles (>=2.1.3)",
+]
+
+[project.urls]
+Homepage = "https://julianschelb.github.io/retexo/"
+Documentation = "https://julianschelb.github.io/retexo/"
+Repository = "https://github.com/julianschelb/retexo"
+Demo = "https://julianschelb.github.io/retexo/demo/"
+Dataset = "https://huggingface.co/datasets/julian-schelb/latin-classical-intertextuality-edit-scripts"
+Changelog = "https://github.com/julianschelb/retexo/blob/main/CHANGELOG.md"
+Issues = "https://github.com/julianschelb/retexo/issues"
+
+[project.optional-dependencies]
+gui = [
+    "gradio (>=5.49.1)",
+]
+lexical = [
+    # The symbolic typer and the lexical evidence look words up in these resources.
+    "cltk (>=1.1.0,<2.0.0) ; python_version < '3.13'",
+    "spacy (>=3.5.0,<4.0.0)",
+    "stanza (>=1.7.0)",
+    "nltk (>=3.8.0)",
+    "gensim (>=4.3.0,<5.0.0)",
+]
+llm = [
+    "anthropic (>=0.30.0)",
+    "peft (>=0.10.0)",
+]
+plots = [
+    "matplotlib (>=3.7.0)",
+]
+dev = [
+    "pytest (>=8.0.0,<10.0.0)",
+    "pytest-cov (>=4.0.0)",
+    "poethepoet (>=0.24.0)",
+    "mkdocs (>=1.5.0)",
+    "mkdocs-material (>=9.0.0)",
+    "mkdocstrings[python] (>=0.24.0)",
+    "ruff (>=0.8.0)",
+    "pre-commit (>=3.5.0)",
+    "build",
+    "twine",
+]
+
+[project.scripts]
+retexo = "retexo.cli:main"
+retexo-gui = "retexo_gui.app:main"
+
+[tool.poetry]
+packages = [
+    {include = "retexo", from = "src"},
+    {include = "retexo_gui", from = "src"},
+]
+
+[build-system]
+requires = ["poetry-core>=2.0.0,<3.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+python_classes = ["Test*"]
+python_functions = ["test_*"]
+addopts = "-v"
+pythonpath = ["src"]
+
+[tool.poe.tasks]
+test = "pytest"
+test-cov = "pytest --cov=retexo --cov-report=term-missing"
+lint = "ruff check src/ tests/"
+docs = "mkdocs serve"
+docs-build = "mkdocs build"
+webapp = "npm --prefix webapp run dev"
+
+# ---------- Ruff ----------
+# The research code was written without a formatter; CI checks for errors only (syntax errors, undefined names).
+
+[tool.ruff]
+target-version = "py310"
+line-length = 120
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = ["E9", "F63", "F7", "F82"]

retexo-0.1.0/src/retexo/__init__.py

@@ -0,0 +1,62 @@
+# retexo/__init__.py
+"""
+Typed edit scripts for Latin intertextual reuse.
+
+Building a variant and checking it round-trips:
+
+```python
+from retexo import VariantBuilder, Scriba
+
+builder = VariantBuilder("uox faucibus haesit")
+builder.keep(0).syn(1, "gutture").keep(2)
+variant, script = builder.build()
+
+Scriba().verify(script, "uox faucibus haesit".split(), variant)   # True
+Scriba().execute(script.invert(), variant)                        # back to the source
+```
+"""
+from __future__ import annotations
+
+__version__ = "0.1.0"
+
+from retexo.core.builder import VariantBuilder
+from retexo.core.oracle import (
+    Aligner,
+    EditPlanOracle,
+    GreedyAligner,
+    OptimalAligner,
+    OracleConfig,
+)
+from retexo.resources import Resources
+from retexo.core.normalize import normalize
+from retexo.operations import (
+    ALL_OPERATIONS,
+    EditOperation,
+    Level,
+    Operation,
+    OperationRegistry,
+    Role,
+)
+from retexo.core.scriba import Scriba, Validity
+from retexo.core.script import CostModel, EditScript
+
+__all__ = [
+    "ALL_OPERATIONS",
+    "Aligner",
+    "EditPlanOracle",
+    "GreedyAligner",
+    "OptimalAligner",
+    "OracleConfig",
+    "Resources",
+    "CostModel",
+    "EditOperation",
+    "EditScript",
+    "Level",
+    "Operation",
+    "OperationRegistry",
+    "Role",
+    "Scriba",
+    "Validity",
+    "VariantBuilder",
+    "normalize",
+]

retexo-0.1.0/src/retexo/aligners/__init__.py

@@ -0,0 +1,27 @@
+# retexo/aligners/__init__.py
+"""Alignment models and the corpus-level and pair-level policies that turn their scores into
+one source per reuse word: the classical pointer-score assignment policies, the bidirectional
+agreement rules, the orthographic sameness predicates, the decoder that derives a full script
+from a typed alignment, the Sinkhorn and whole-script DP variants, and the contextual embedding
+aligner."""
+from __future__ import annotations
+
+from retexo.aligners.agreement import AgreementDecoder, PairSwap
+from retexo.aligners.aligner import ContextualAligner
+from retexo.aligners.assignment import AssignmentPolicy, Reranker
+from retexo.aligners.decode import ScriptDecoder
+from retexo.aligners.global_decode import GlobalDecoder
+from retexo.aligners.sameness import SamenessPolicy
+from retexo.aligners.sinkhorn import SinkhornBalancer
+
+__all__ = [
+    "AgreementDecoder",
+    "AssignmentPolicy",
+    "ContextualAligner",
+    "GlobalDecoder",
+    "PairSwap",
+    "Reranker",
+    "SamenessPolicy",
+    "ScriptDecoder",
+    "SinkhornBalancer",
+]

retexo-0.1.0/src/retexo/aligners/agreement.py

@@ -0,0 +1,181 @@
+# retexo/aligners/agreement.py
+"""
+Bidirectional-agreement decoding rules: score both directions, keep what both accept.
+
+Renamed from the preliminary experiment module E29; kept under its own name,
+unchanged, because ``retexo.baselines.decoder`` imports it directly and
+several implementation notes say explicitly to reuse it verbatim.
+
+Our pointer scores reuse -> source and its null is a scalar per reuse word.
+awesome-align's null is a test on two directions: a link exists only if the
+reuse word picks the source word *and* the source word picks it back.
+SimAlign's mutual argmax is the same test without a threshold; its entropy
+rule is a third signal. All three work on two sets of rows -- the forward
+rows the pointer scores, and the reverse rows a swapped-sides pass produces --
+and need no training.
+
+Rows are the pointer's score-row shape: per reuse word, [(source, p) ...] best
+first, with the null as source -1. Reverse rows are the same shape with the
+roles swapped: per *source* word, [(reuse, p) ...].
+"""
+
+from __future__ import annotations
+
+import math
+from typing import ClassVar, Dict, List, Sequence, Tuple
+
+from retexo.formulations.change_detector import ChangeExample
+
+Row = List[Tuple[int, float]]
+
+
+class PairSwap:
+    """Building a pair with the roles reversed, for the reverse-direction pass
+    and for symmetric training.
+
+    Example:
+        ```python
+        reverse_pair = PairSwap.example(example)            # untrained: p(reuse word | source word)
+        reverse_training_pair = PairSwap.labelled(example)   # a training pair with inverted labels
+        ```
+    """
+
+    #: relations that change name when the sides change
+    INVERT: ClassVar[Dict[str, str]] = {"HYPER": "HYPO", "HYPO": "HYPER", "SPLIT": "MERGE", "MERGE": "SPLIT"}
+
+    @staticmethod
+    def example(example: ChangeExample) -> ChangeExample:
+        """The pair with the roles exchanged: the source passage sits in the reuse
+        slot, so the model scores p(reuse word | source word)."""
+        s, t = list(example.target_tokens), list(example.source_tokens)   # swapped on purpose
+        return ChangeExample(source_tokens=s, target_tokens=t, labels=[0] * len(t),
+                             operations=["COPY"] * len(t), n_operations=0,
+                             source_labels=[0] * len(s), source_operations=["COPY"] * len(s),
+                             alignments=[-1] * len(t), fine_operations=["INS"] * len(t),
+                             frame_labels=[0] * len(t), link_features=[None] * len(t))
+
+    @classmethod
+    def labelled(cls, example: ChangeExample) -> ChangeExample:
+        """A *training* pair with the sides exchanged and its labels inverted.
+
+        One-to-one links invert exactly; HYPER/HYPO and SPLIT/MERGE change name;
+        a gold lexical change of unknown kind ("?") stays unknown; the former reuse
+        side's inserted words become deleted source words; frames do not exist on
+        a source passage, so the new reuse side carries none. Evidence grids are
+        not transposed (the featurizer is not symmetric) -- recompute them."""
+        old_s, old_t = list(example.source_tokens), list(example.target_tokens)
+        links = example.alignments or [-1] * len(old_t)
+        fine = example.fine_operations or ["INS"] * len(old_t)
+        new_s, new_t = old_t, old_s
+        new_links = [-1] * len(new_t)
+        new_fine = ["INS"] * len(new_t)
+        for t, s in enumerate(links):
+            if s is not None and 0 <= s < len(new_t) and new_links[s] < 0:
+                new_links[s] = t
+                tag = fine[t] if t < len(fine) else "?"
+                new_fine[s] = cls.INVERT.get(tag, tag)
+        consumed = {t for t in new_links if t >= 0}
+        new_source_labels = [0 if t in consumed else 1 for t in range(len(new_s))]
+        coarse = ["COPY" if (k == "NOP") else ("INS" if k == "INS" else "SUBST") for k in new_fine]
+        ex = ChangeExample(source_tokens=new_s, target_tokens=new_t,
+                           labels=[0 if op == "COPY" else 1 for op in coarse], operations=coarse,
+                           n_operations=sum(1 for op in coarse if op != "COPY"),
+                           source_labels=new_source_labels,
+                           source_operations=["DEL" if d else "COPY" for d in new_source_labels],
+                           alignments=new_links, fine_operations=new_fine,
+                           frame_labels=[0] * len(new_t), link_features=[None] * len(new_t))
+        object.__setattr__(ex, "swapped_from", True)
+        return ex
+
+
+class AgreementDecoder:
+    """Bidirectional-agreement rules, each returning restricted forward rows for
+    the Hungarian assignment.
+
+    Example:
+        ```python
+        kept = AgreementDecoder.mutual_argmax(rows, reverse_rows)
+        kept = AgreementDecoder.intersect(rows, reverse_rows, c=0.4)
+        ```
+    """
+
+    @staticmethod
+    def prob(row: Row, index: int) -> float:
+        return dict(row).get(index, 0.0)
+
+    @staticmethod
+    def top(row: Row) -> int:
+        """The best entry, the null (-1) included."""
+        return row[0][0] if row else -1
+
+    @classmethod
+    def reverse_links(cls, rows_rev: Sequence[Row], n_reuse: int) -> List[int]:
+        """The reverse direction read as an aligner of the reuse side: reuse word t
+        links to the source word s whose best pick is t (first such s), else -1."""
+        links = [-1] * n_reuse
+        for s, row in enumerate(rows_rev):
+            t = cls.top(row)
+            if 0 <= t < n_reuse and links[t] < 0:
+                links[t] = s
+        return links
+
+    @staticmethod
+    def entropy(row: Row) -> float:
+        ps = [p for _, p in row if p > 1e-12]
+        z = sum(ps) or 1.0
+        return -sum(p / z * math.log(p / z) for p in ps)
+
+    @classmethod
+    def intersect(cls, rows: Sequence[Row], rows_rev: Sequence[Row], c: float) -> List[Row]:
+        """awesome-align: keep (t, s) iff p(s|t) > c and p(t|s) > c. A word whose
+        every candidate fails keeps only its null."""
+        out = []
+        for t, row in enumerate(rows):
+            kept = [(s, p) for s, p in row
+                    if s >= 0 and p > c and s < len(rows_rev) and cls.prob(rows_rev[s], t) > c]
+            out.append(sorted(kept + [(-1, cls.prob(row, -1))], key=lambda x: -x[1]))
+        return out
+
+    @classmethod
+    def mutual_argmax(cls, rows: Sequence[Row], rows_rev: Sequence[Row]) -> List[Row]:
+        """SimAlign Argmax: keep (t, s) iff s is t's best and t is s's best."""
+        out = []
+        for t, row in enumerate(rows):
+            s = cls.top(row)
+            kept = [(s, cls.prob(row, s))] if s >= 0 and s < len(rows_rev) and cls.top(rows_rev[s]) == t else []
+            out.append(sorted(kept + [(-1, cls.prob(row, -1))], key=lambda x: -x[1]))
+        return out
+
+    @classmethod
+    def entropy_filter(cls, rows: Sequence[Row], rows_rev: Sequence[Row], tau: float) -> List[Row]:
+        """SimAlign's null: drop every link of a reuse word whose row entropy, and of
+        a source word whose column entropy, are both above tau (normalised by log n)."""
+        out = []
+        h_col = [cls.entropy(r) / max(math.log(max(len(r), 2)), 1e-9) for r in rows_rev]
+        for t, row in enumerate(rows):
+            h_row = cls.entropy(row) / max(math.log(max(len(row), 2)), 1e-9)
+            kept = [(s, p) for s, p in row if s >= 0
+                    and min(h_row, h_col[s] if s < len(h_col) else 1.0) <= tau]
+            out.append(sorted(kept + [(-1, cls.prob(row, -1))], key=lambda x: -x[1]))
+        return out
+
+    @classmethod
+    def null_scale(cls, rows: Sequence[Row], k: float) -> List[Row]:
+        """Our null re-weighted by k and renormalised (E18's move, at the raw pointer)."""
+        out = []
+        for row in rows:
+            scaled = [(s, p * (k if s < 0 else 1.0)) for s, p in row]
+            z = sum(p for _, p in scaled) or 1.0
+            out.append(sorted([(s, p / z) for s, p in scaled], key=lambda x: -x[1]))
+        return out
+
+    @classmethod
+    def compose(cls, *restricted: Sequence[Row]) -> List[Row]:
+        """Keep a candidate only if every rule kept it; the null is the first rule's."""
+        out = []
+        for rows in zip(*restricted):
+            keep = set.intersection(*[{s for s, _ in r if s >= 0} for r in rows])
+            first = rows[0]
+            out.append(sorted([(s, p) for s, p in first if s in keep] + [(-1, cls.prob(first, -1))],
+                              key=lambda x: -x[1]))
+        return out