biblealignlib 0.1.4__tar.gz

biblealignlib-0.1.4/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Clear.Bible
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

biblealignlib-0.1.4/LICENSE.md

@@ -0,0 +1,92 @@
+# Bible Word Alignments
+
+We license all of our own code under an MIT license and all of our own
+data under CC-BY.  For details, see sections below for [Code](#code) and [Data](#data).
+
+## Code
+
+Code for this project (`../bible_alignments`) is copyright (c) 2023 by
+[Clear Bible, Inc](http://www.clear.bible) and is licensed under the
+terms of the MIT License.
+
+### MIT License
+
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+“Software”), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+## Data
+
+[Bible Word Alignments](https://github.com/Clear-Bible/Alignments) © 2022 by [Clear Bible, Inc](http://www.clear.bible) is licensed under [CC BY 4.0 ](http://creativecommons.org/licenses/by/4.0/).
+
+These datasets include:
+
+1. Alignment files (`../data/alignments`) derived from Clear Bible's data.
+2. Source text files (`../data/sources`) derived from Clear Bible's
+   data. Note that any copyright-protected text has been stripped out.
+3. Target text files (`../data/targets`) derived from Clear Bible's
+   data. Note that any copyright-protected text has been stripped out.
+4. Names files (`../data/targets`) derived from alignment data.
+
+Source text files include data from:
+
+* Westminster Leningrad Codex - the somewhat informal license states
+  that "All biblical Hebrew text, in any format, may be viewed or
+  copied without restriction."
+
+Target text files include data from:
+
+* The text for the Chinese Union Version (Simplified) is in the public
+  domain. The Chinese Union Version with Modern Punctuation
+  (Simplified) (`CUVMP`) is a derivative work, and is Copyright © 2011
+  Global Bible Initiative / © 2011 全球圣经促进会 and in the public
+  domain.
+* Young's Literal Translation (`YLT`), by Robert Young (1862, 1887, 1898),
+  which is in the public domain.
+
+The repository also includes data on strategic languages for Bible
+translation (`../data/languages`) from the [ETEN Innovation
+Lab](https://dev.lab.eten.bible/).
+
+### License
+
+#### Creative Commons Attribution 4.0 International (CC BY 4.0)
+
+This is a human-readable summary of (and not a substitute for) the [license](http://creativecommons.org/licenses/by/4.0/).
+
+##### You are free to:
+
+ * **Share** — copy and redistribute the material in any medium or format
+ * **Adapt** — remix, transform, and build upon the material
+for any purpose, even commercially.
+
+The licensor cannot revoke these freedoms as long as you follow the license terms.
+
+##### Under the following terms:
+
+ * **Attribution** — You must attribute the work as follows: "MACULA Greek Linguistic Datasets, available at https://github.com/Clear-Bible/macula-greek/". You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
+
+**No additional restrictions** — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
+
+##### Notices:
+
+You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.
+
+No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.
+

biblealignlib-0.1.4/PKG-INFO

@@ -0,0 +1,59 @@
+Metadata-Version: 2.1
+Name: biblealignlib
+Version: 0.1.4
+Summary: Code for managing Word-level alignments for Bibles, including both automatic alignments and manually corrected alignments.
+Home-page: https://github.com/Clear-Bible/biblealignlib
+License: MIT
+Keywords: Bible,alignment,Bible alignment
+Author: Sean Boisen
+Author-email: sean.boisen@biblica.com
+Requires-Python: >=3.10,<3.12
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Religion
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Religion
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Dist: altair (>=5.5.0,<6.0.0)
+Requires-Dist: biblelib (>=0.3.17,<0.4.0)
+Requires-Dist: jupyter-server-ydoc (>=1.1.0,<2.0.0)
+Requires-Dist: jupyterlab (>=4.3.3,<5.0.0)
+Requires-Dist: pandas (>=2.2.3,<3.0.0)
+Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
+Requires-Dist: regex (>=2024.11.6,<2025.0.0)
+Requires-Dist: unicodecsv (>=0.14.1,<0.15.0)
+Project-URL: Repository, https://github.com/Clear-Bible/biblealignlib
+Description-Content-Type: text/markdown
+
+# biblealignlib
+
+Biblica's code for working with Bible alignment data from
+https://github.com/Clear-Bible/Alignments .
+
+Currently private but we should move toward a future where it's
+public. 
+
+## Installing extra dependencies
+
+### eflomal
+`eflomal` is specified as an extra, so it is not installed with `poetry install`.
+
+On macOS, you may need to install additional dependencies before installing:
+
+```
+brew install llvm libomp
+```
+
+You'll need to override the `CFLAGS` and `LDFLAGS` environment variables before installing `eflomal`.
+
+```
+poetry shell
+export CFLAGS="-Xpreprocessor -fopenmp -I${HOMEBREW_PREFIX}/opt/libomp/include -Ofast -march=native -Wall --std=gnu99 -Wno-unused-function -g"
+export LDFLAGS="-Xpreprocessor -fopenmp -L${HOMEBREW_PREFIX}/opt/libomp/lib -lm -lomp"
+poetry install -E eflomal
+```
+
+

biblealignlib-0.1.4/README.md

@@ -0,0 +1,28 @@
+# biblealignlib
+
+Biblica's code for working with Bible alignment data from
+https://github.com/Clear-Bible/Alignments .
+
+Currently private but we should move toward a future where it's
+public. 
+
+## Installing extra dependencies
+
+### eflomal
+`eflomal` is specified as an extra, so it is not installed with `poetry install`.
+
+On macOS, you may need to install additional dependencies before installing:
+
+```
+brew install llvm libomp
+```
+
+You'll need to override the `CFLAGS` and `LDFLAGS` environment variables before installing `eflomal`.
+
+```
+poetry shell
+export CFLAGS="-Xpreprocessor -fopenmp -I${HOMEBREW_PREFIX}/opt/libomp/include -Ofast -march=native -Wall --std=gnu99 -Wno-unused-function -g"
+export LDFLAGS="-Xpreprocessor -fopenmp -L${HOMEBREW_PREFIX}/opt/libomp/lib -lm -lomp"
+poetry install -E eflomal
+```
+

biblealignlib-0.1.4/biblealignlib/__init__.py

@@ -0,0 +1,110 @@
+"""Internal-only code for working with alignment data."""
+
+from enum import Enum
+import os
+from pathlib import Path
+import re
+
+import dotenv
+
+from .strongs import normalize_strongs
+
+# it would be nice to import symbols from burrito and autoalign here:
+# but i don't know how to avoid circular imports, when that codes also
+# imports from biblealignlib
+
+# set path variables. These assume you have a .env file that locates
+# the directory where Clear-Bible repositories are located, like
+#
+# CLEARROOT=/Users/sboisen/git/Clear-Bible
+#
+# use an environment variable if
+if not dotenv.load_dotenv():
+    print("No .env file found")
+clearrootenvar = os.getenv("CLEARROOT")
+if clearrootenvar:
+    CLEARROOT = Path(clearrootenvar)
+else:
+    CLEARROOT = Path.home() / "git/Clear-Bible"
+    print(f"No environment variable for CLEARROOT: assuming {CLEARROOT}")
+
+# for loading published data. Alignments are here under language
+ALIGNMENTSDATA = CLEARROOT / "Alignments/data"
+# for loading published source TSVs
+SOURCES = ALIGNMENTSDATA / "sources"
+
+CANONIDS = {
+    "nt",
+    "ot",
+    # meaning the entire 66 book corpus
+    "protestant",
+}
+
+
+VERSIFICATIONIDS: set[str] = {
+    "eng",
+    "org",
+    "rso",
+    # not yet implemented
+    # "ethiopian_custom", "lxx", "rsc", "vul"
+}
+
+
+class SourceidEnum(str, Enum):
+    """Valid source identifiers."""
+
+    BGNT = "BGNT"
+    NA27 = "NA27"
+    NA28 = "NA28"
+    SBLGNT = "SBLGNT"
+    WLC = "WLC"
+    WLCM = "WLCM"
+
+    @property
+    def canon(self) -> str:
+        """Return 'ot' or 'nt' for the canon."""
+        if self.value in ["WLC", "WLCM"]:
+            return "ot"
+        elif self.value in ["BGNT", "NA27", "NA28", "SBLGNT"]:
+            return "nt"
+        else:
+            raise ValueError(f"Unknown error in SourceidEnum.canon for {self.value}")
+
+    # need to add DC, probably others down the road
+    @staticmethod
+    def get_canon(sourceid: str) -> str:
+        """Return a canon string for recognized sources, else 'X'."""
+        try:
+            srcenum = SourceidEnum(sourceid)
+            return srcenum.canon
+        except ValueError:
+            # unrecognized source
+            return "X"
+
+
+def get_canonid(bcv: str) -> str:
+    """Return nt/ot for a BCVish string.
+
+    Simple string matching on the book portion of an identifier, so
+    works for books, chapters, verses and full BCVWPID identifiers.
+
+    """
+    otcanonre = re.compile(r"^[0-3][0-9]")
+    ntcanonre = re.compile(r"^[4-6][0-9]")
+    # don't include 67-69
+    notntcanonre = re.compile(r"^6[7-9]")
+    if otcanonre.match(bcv):
+        return "ot"
+    elif ntcanonre.match(bcv) and not notntcanonre.match(bcv):
+        return "nt"
+    else:
+        raise ValueError(f"Invalid BCVish id value: {bcv}")
+
+
+__all__ = [
+    "CLEARROOT",
+    "SOURCES",
+    "SourceidEnum",
+    # strongs
+    "normalize_strongs",
+]

biblealignlib-0.1.4/biblealignlib/autoalign/Score.py

@@ -0,0 +1,144 @@
+"""Manage scores for alignment data."""
+
+from dataclasses import dataclass, field
+from typing import Any, Optional
+
+from biblelib.word import BCVID
+
+from biblealignlib.burrito import Source, Target, VerseData
+
+
+def precision(true_positives: int, false_positives: int) -> float:
+    denom = true_positives + false_positives
+    return true_positives / denom if denom else 0
+
+
+def recall(true_positives: int, false_negatives: int) -> float:
+    denom = true_positives + false_negatives
+    return true_positives / denom if denom else 0
+
+
+def f1(recall: float, precision: float) -> float:
+    denom = precision + recall
+    return ((2 * precision * recall) / denom) if denom else 0
+
+
+@dataclass
+class _BaseScore:
+    """Manage base scoring metrics."""
+
+    identifier: str = ""
+    true_positives: int = 0
+    # true_negatives: int = 0
+    false_positives: int = 0
+    false_negatives: int = 0
+    precision: float = 0.0
+    recall: float = 0.0
+    f1: float = 0.0
+    aer: float = 0.0
+
+    def __repr__(self) -> str:
+        """Return a string representation of the Score."""
+        return f"<{self.__class__.__name__}: {self.identifier}>"
+
+    def compute_metrics(self) -> None:
+        """Compute various metrics."""
+        self.precision = precision(self.true_positives, self.false_positives)
+        self.aer = 1 - self.precision
+        self.recall = recall(self.true_positives, self.false_negatives)
+        self.f1 = f1(self.recall, self.precision)
+
+    # should use summary_dict here
+    def summary(self, width: int = 4, brief: bool = True) -> str:
+        """Return summary metrics."""
+        plabel = "P" if brief else "Precision"
+        rlabel = "R" if brief else "Recall"
+        return f"{self.identifier}: AER={self.aer:.{width}f}\t{plabel}={self.precision:.{width}f}\t{rlabel}={self.recall:.{width}f}\tF1={self.f1:.{width}f}"
+
+    def summary_dict(self, width: int = 4) -> dict[str, str]:
+        """Return a dict with summary scores."""
+        return {
+            "AER": f"{self.aer:.{width}f}",
+            "F1": f"{self.f1:.{width}f}",
+            "Precision": f"{self.precision:.{width}f}",
+            "Recall": f"{self.recall:.{width}f}",
+        }
+
+    def asdict(self, ndigits=3) -> dict[str, Any]:
+        """Return a dict usable as a dataframe row."""
+        scoredict = {
+            # this _should_ always be a BCV
+            "Identifier": self.identifier,
+            # just the verse index
+            "Verse": self.identifier[5:],
+            "Chapter": self.identifier[:5],
+            "Book": self.identifier[:2],
+            "Reference": BCVID(self.identifier).to_usfm(),
+            "AER": round(self.aer, ndigits),
+            "F1": round(self.f1, ndigits),
+            "Precision": round(self.precision, ndigits),
+            "Recall": round(self.recall, ndigits),
+        }
+        return scoredict
+
+
+@dataclass(repr=False)
+class VerseScore(_BaseScore):
+    """Manage scoring data for a verse."""
+
+    # not really optional, but dataclass inheritance requires this
+    reference: Optional[VerseData] = None
+    hypothesis: Optional[VerseData] = None
+    # computed
+    n_sources: int = 0
+    n_targets: int = 0
+    referencepairs: list[tuple[Source, Target]] = field(init=False, default_factory=list)
+    hypothesispairs: list[tuple[Source, Target]] = field(init=False, default_factory=list)
+
+    def __post_init__(self) -> None:
+        """Compute values on initialization."""
+        self.identifier = self.reference.bcvid
+        self.n_sources = len(self.reference.sources)
+        self.n_targets = len(self.reference.targets)
+        # decompose into pairs of source and target indices
+        self._get_pairs()
+        # set operations on pairs: no partial credit
+        self.true_positives = len(set(self.referencepairs) & set(self.hypothesispairs))
+        self.false_positives = len(set(self.hypothesispairs) - set(self.referencepairs))
+        self.false_negatives = len(set(self.referencepairs) - set(self.hypothesispairs))
+        # sets values for P, R, F1, AER
+        self.compute_metrics()
+
+    def _get_pairs(self) -> None:
+        """Populate reference/hypothesispairs."""
+        # these are like pharaoh: tokens are repeated for multiple alignments
+        self.referencepairs = self.reference.get_pairs()
+        self.hypothesispairs = self.hypothesis.get_pairs()
+
+
+@dataclass(repr=False)
+class EssentialVerseScore(VerseScore):
+    """Like VerseScore but only for essential alignments."""
+
+    def _get_pairs(self) -> None:
+        """Populate reference/hypothesispairs."""
+        # these are like pharaoh: tokens are repeated for multiple alignments
+        self.referencepairs = self.reference.get_pairs(essential=True)
+        self.hypothesispairs = self.hypothesis.get_pairs(essential=True)
+
+
+@dataclass(repr=False)
+class GroupScore(_BaseScore):
+    """Manage scoring data for a group of verses."""
+
+    verse_scores: list[VerseScore] = field(default_factory=list)
+
+    def __post_init__(self) -> None:
+        """Compute values on initialization."""
+        assert self.identifier, "Must provide identifier"
+        assert self.verse_scores, "Must provide verse_scores."
+
+        self.true_positives: int = sum(v.true_positives for v in self.verse_scores)
+        self.false_positives: int = sum(v.false_positives for v in self.verse_scores)
+        self.false_negatives: int = sum(v.false_negatives for v in self.verse_scores)
+        self.compute_metrics()

biblealignlib-0.1.4/biblealignlib/autoalign/__init__.py

@@ -0,0 +1,23 @@
+from biblealignlib import CLEARROOT, SOURCES
+
+# when it gets fixed
+# from .eflomal import Eflomal
+from .mapper import PharaohMapper
+from .reader import PharaohReader
+from .scorer import Scorer
+from .writer import PharaohWriter
+
+__all__ = [
+    "CLEARROOT",
+    "SOURCES",
+    # # eflomal
+    # "Eflomal",
+    # mapper
+    "PharaohMapper",
+    # reader
+    "PharaohReader",
+    # scorer
+    "Scorer",
+    # writer
+    "PharaohWriter",
+]

biblealignlib-0.1.4/biblealignlib/autoalign/corpusmapping.py

@@ -0,0 +1,73 @@
+"""Manages corpus data for auto alignment."""
+
+from dataclasses import dataclass, field
+
+from biblealignlib.burrito import (
+    BaseToken,
+    Source,
+    Target,
+)
+
+
+@dataclass
+class CorpusMapping:
+    """Map corpus instances to pharaoh-data for a single verse correspondence.
+
+    bcv is based on source versification: targets with different
+    versifications should be mapped to comparable source verses in the
+    TSV.
+
+    Example: Target:01031005007 corresponds to Source:01032001
+
+    """
+
+    # BCV-format verse reference
+    bcv: str
+    # Source instances and their pharaoh indices
+    source_pairs: list[tuple[Source, int]] = field(default_factory=list)
+    # Target instances and their pharaoh indices
+    target_pairs: list[tuple[Target, int]] = field(default_factory=list)
+    _typeattrs: tuple = ("sources", "targets")
+    # these values computed in post_init
+    # dict: index -> Token
+    sourceindexmap: dict[int, BaseToken] = field(default_factory=dict)
+    targetindexmap: dict[int, BaseToken] = field(default_factory=dict)
+    # dict: Token -> index
+    sourcetokenmap: dict[BaseToken, int] = field(default_factory=dict)
+    targettokenmap: dict[BaseToken, int] = field(default_factory=dict)
+
+    def __post_init__(self) -> None:
+        """Compute values after initialization."""
+        self.sourceindexmap = {index: item for item, index in self.source_pairs}
+        self.targetindexmap = {index: item for item, index in self.target_pairs}
+        self.sourcetokenmap = dict(self.source_pairs)
+        self.targettokenmap = dict(self.target_pairs)
+
+    def __repr__(self) -> str:
+        """Return a string representation of the CorpusMapping."""
+        return f"<CorpusMapping: {self.bcv}>"
+
+    def tokenids(self, typeattr: str) -> list[str]:
+        """Return the list of corpus token ids for typeattr."""
+        assert typeattr in self._typeattrs, f"typeattr should be one of {self._typeattrs}"
+        pairs = self.source_pairs if typeattr == "sources" else self.target_pairs
+        return [corpus.id for corpus, _ in pairs]
+
+    def indices(self, typeattr: str) -> list[str]:
+        """Return the list of pharaoah indices for typeattr."""
+        assert typeattr in self._typeattrs, f"typeattr should be one of {self._typeattrs}"
+        pairs = self.source_pairs if typeattr == "sources" else self.target_pairs
+        return [index for _, index in pairs]
+
+    def tokentexts(self, typeattr: str) -> list[str]:
+        """Return the list of corpus token texts for typeattr."""
+        assert typeattr in self._typeattrs, f"typeattr should be one of {self._typeattrs}"
+        pairs = self.source_pairs if typeattr == "sources" else self.target_pairs
+        return [corpus.text for corpus, _ in pairs]
+
+    def display(self, typeattr: str) -> None:
+        """Print out the id and text pairs for debugging."""
+        assert typeattr in self._typeattrs, f"typeattr should be one of {self._typeattrs}"
+        pairs = self.source_pairs if typeattr == "sources" else self.target_pairs
+        for token, index in pairs:
+            print(f"{index}: {token.id}, {token.text}")