biblealignlib 0.1.0__tar.gz

biblealignlib-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Clear.Bible
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

biblealignlib-0.1.0/LICENSE.md

@@ -0,0 +1,92 @@
+# Bible Word Alignments
+
+We license all of our own code under an MIT license and all of our own
+data under CC-BY.  For details, see sections below for [Code](#code) and [Data](#data).
+
+## Code
+
+Code for this project (`../bible_alignments`) is copyright (c) 2023 by
+[Clear Bible, Inc](http://www.clear.bible) and is licensed under the
+terms of the MIT License.
+
+### MIT License
+
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+“Software”), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+## Data
+
+[Bible Word Alignments](https://github.com/Clear-Bible/Alignments) © 2022 by [Clear Bible, Inc](http://www.clear.bible) is licensed under [CC BY 4.0 ](http://creativecommons.org/licenses/by/4.0/).
+
+These datasets include:
+
+1. Alignment files (`../data/alignments`) derived from Clear Bible's data.
+2. Source text files (`../data/sources`) derived from Clear Bible's
+   data. Note that any copyright-protected text has been stripped out.
+3. Target text files (`../data/targets`) derived from Clear Bible's
+   data. Note that any copyright-protected text has been stripped out.
+4. Names files (`../data/targets`) derived from alignment data.
+
+Source text files include data from:
+
+* Westminster Leningrad Codex - the somewhat informal license states
+  that "All biblical Hebrew text, in any format, may be viewed or
+  copied without restriction."
+
+Target text files include data from:
+
+* The text for the Chinese Union Version (Simplified) is in the public
+  domain. The Chinese Union Version with Modern Punctuation
+  (Simplified) (`CUVMP`) is a derivative work, and is Copyright © 2011
+  Global Bible Initiative / © 2011 全球圣经促进会 and in the public
+  domain.
+* Young's Literal Translation (`YLT`), by Robert Young (1862, 1887, 1898),
+  which is in the public domain.
+
+The repository also includes data on strategic languages for Bible
+translation (`../data/languages`) from the [ETEN Innovation
+Lab](https://dev.lab.eten.bible/).
+
+### License
+
+#### Creative Commons Attribution 4.0 International (CC BY 4.0)
+
+This is a human-readable summary of (and not a substitute for) the [license](http://creativecommons.org/licenses/by/4.0/).
+
+##### You are free to:
+
+ * **Share** — copy and redistribute the material in any medium or format
+ * **Adapt** — remix, transform, and build upon the material
+for any purpose, even commercially.
+
+The licensor cannot revoke these freedoms as long as you follow the license terms.
+
+##### Under the following terms:
+
+ * **Attribution** — You must attribute the work as follows: "MACULA Greek Linguistic Datasets, available at https://github.com/Clear-Bible/macula-greek/". You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
+
+**No additional restrictions** — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
+
+##### Notices:
+
+You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.
+
+No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.
+

biblealignlib-0.1.0/PKG-INFO

@@ -0,0 +1,30 @@
+Metadata-Version: 2.1
+Name: biblealignlib
+Version: 0.1.0
+Summary: Code for managing Word-level alignments for Bibles, including both automatic alignments and manually corrected alignments.
+Home-page: https://github.com/Clear-Bible/biblealignlib
+License: MIT
+Keywords: Bible,alignment,Bible alignment
+Author: Sean Boisen
+Author-email: sean.boisen@biblica.com
+Requires-Python: >=3.10,<3.12
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Religion
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Religion
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Dist: biblelib (>=0.3.17,<0.4.0)
+Requires-Dist: jupyterlab (>=4.3.3,<5.0.0)
+Requires-Dist: pandas (>=2.2.3,<3.0.0)
+Requires-Dist: regex (>=2024.11.6,<2025.0.0)
+Requires-Dist: unicodecsv (>=0.14.1,<0.15.0)
+Project-URL: Repository, https://github.com/Clear-Bible/biblealignlib
+Description-Content-Type: text/markdown
+
+# biblealignlib
+Code for working with Bible alignment data
+

biblealignlib-0.1.0/README.md

@@ -0,0 +1,2 @@
+# biblealignlib
+Code for working with Bible alignment data

biblealignlib-0.1.0/biblealignlib/__init__.py

@@ -0,0 +1,93 @@
+"""Internal-only code for working with alignment data."""
+
+from enum import Enum
+from pathlib import Path
+import re
+
+from .strongs import normalize_strongs
+
+
+ROOT = Path(__file__).parent
+DATAPATH = ROOT / "data"
+SRCPATH = ROOT / "src"
+
+GRAPECITYDIR = ROOT.parent / "grapecity-alignments"
+# for output
+ALIGNMENTSROOT = ROOT.parent / "Alignments"
+
+CANONIDS = {
+    "nt",
+    "ot",
+    # meaning the entire 66 book corpus
+    "protestant",
+}
+
+
+VERSIFICATIONIDS: set[str] = {
+    "eng",
+    "org",
+    "rso",
+    # not yet implemented
+    # "ethiopian_custom", "lxx", "rsc", "vul"
+}
+
+
+class SourceidEnum(str, Enum):
+    """Valid source identifiers."""
+
+    BGNT = "BGNT"
+    NA27 = "NA27"
+    NA28 = "NA28"
+    SBLGNT = "SBLGNT"
+    WLC = "WLC"
+    WLCM = "WLCM"
+
+    @property
+    def canon(self) -> str:
+        """Return 'ot' or 'nt' for the canon."""
+        if self.value in ["WLC", "WLCM"]:
+            return "ot"
+        elif self.value in ["BGNT", "NA27", "NA28", "SBLGNT"]:
+            return "nt"
+        else:
+            raise ValueError(f"Unknown error in SourceidEnum.canon for {self.value}")
+
+    # need to add DC, probably others down the road
+    @staticmethod
+    def get_canon(sourceid: str) -> str:
+        """Return a canon string for recognized sources, else 'X'."""
+        try:
+            srcenum = SourceidEnum(sourceid)
+            return srcenum.canon
+        except ValueError:
+            # unrecognized source
+            return "X"
+
+
+def get_canonid(bcv: str) -> str:
+    """Return nt/ot for a BCVish string.
+
+    Simple string matching on the book portion of an identifier, so
+    works for books, chapters, verses and full BCVWPID identifiers.
+
+    """
+    otcanonre = re.compile(r"^[0-3][0-9]")
+    ntcanonre = re.compile(r"^[4-6][0-9]")
+    # don't include 67-69
+    notntcanonre = re.compile(r"^6[7-9]")
+    if otcanonre.match(bcv):
+        return "ot"
+    elif ntcanonre.match(bcv) and not notntcanonre.match(bcv):
+        return "nt"
+    else:
+        raise ValueError(f"Invalid BCVish id value: {bcv}")
+
+
+__all__ = [
+    "ROOT",
+    "DATAPATH",
+    "SRCPATH",
+    "SourceidEnum",
+    # strongs
+    "normalize_strongs",
+]

biblealignlib-0.1.0/biblealignlib/burrito/AlignmentGroup.py

@@ -0,0 +1,420 @@
+"""Manage data for an alignment record.
+
+This defines AlignmentGroup and supporting dataclasses.
+
+This implements Scripture Burrito Alignment Standard, v 0.3,
+https://docs.google.com/document/d/1zR5gsrm3gIoNiHVBlWz5_BBw3N-Ew1-4M5rMsFrPzSw/.
+
+The information model allows some values at multiple levels due to
+'hoisting'. This opinionated code:
+- defines attributes at the lowest relevant class
+- use maximal hoisting for serialization unless overridded
+
+While this aims at generality, the main application supported here is
+publishing Bible alignment data. There may therefore be aspects of the
+spec that are not supported by this code.
+
+"""
+
+from dataclasses import dataclass, field, fields
+import datetime as dt
+from functools import total_ordering
+from itertools import groupby
+from typing import Any, Optional
+
+from biblelib.word import bcvwpid
+
+from biblealignlib import SourceidEnum
+
+from .AlignmentType import TranslationType
+from .source import macula_prefixer
+
+
+# hoisting means this can be defined at several different levels, so
+# called out as a separate class
+@dataclass
+class Document:
+    """Manage data for an alignment document."""
+
+    # best practices
+    # - for source documents: a standard identifier like 'NA28' or
+    #   "WLC"
+    # - for target documents: a Version Abbreviation value from
+    #   digitalbiblelibrary.org is a good choice.
+    docid: str
+    scheme: str = "BCVWP"
+    # only set if a source ID
+    sourceid: Optional[SourceidEnum] = None
+
+    def __post_init__(self) -> None:
+        """Compute values after initialization."""
+        try:
+            # set is_source if in the standard list
+            self.sourceid = SourceidEnum(self.docid)
+        except ValueError:
+            self.sourceid = None
+            # downgrade BCVWP to BCVW: no subword indices if not source
+            if self.scheme == "BCVWP":
+                self.scheme = "BCVW"
+
+    def asdict(self) -> dict[str, Any]:
+        """Return a dict of values suitable for serialization."""
+        return {"docid": self.docid, "scheme": self.scheme}
+
+
+@dataclass(order=True)
+class AlignmentReference:
+    """Manage data for an alignment reference."""
+
+    # perhaps over-engineered, but Document data can occur at multiple
+    # levels in serialization.
+    document: Document
+    # selectors identify tokens or other units in a document. For most
+    # alignment purposes, these are token identifiers.
+    selectors: list[str]
+
+    def __post_init__(self) -> None:
+        """Compute values after initialization."""
+        # no good for reading alignment hub data
+        # assert bool(self.selectors), "Selectors must not be empty."
+        self.selectors = sorted(self.selectors)
+
+    def __repr__(self) -> str:
+        """Return a printed representation."""
+        return f"<{self.docid}: {self.selectors}>"
+
+    @property
+    def docid(self) -> str:
+        """Return the docid from document, for convenience."""
+        return self.document.docid
+
+    @property
+    def scheme(self) -> str:
+        """Return the scheme from document, for convenience."""
+        return self.document.scheme
+
+    @property
+    def incomplete(self) -> bool:
+        """True if any selectors are MISSING."""
+        return any((sel == "MISSING") for sel in self.selectors)
+
+    def asdict(self, hoist: bool = True) -> dict[str, Any]:
+        """Return a dict of values suitable for serialization.
+
+        With hoist, omit docid and scheme from document, assuming
+        they'll be specified 'higher up'.
+
+        """
+        refdict: dict[str, Any] = {"selectors": self.selectors}
+        if not hoist:
+            refdict.update({"docid": self.docid, "scheme": self.scheme})
+        return refdict
+
+
+@dataclass
+class Metadata:
+    """Contains metadata for alignment records.
+
+    While all attributes are optional, the attributes creator and
+    created are strongly encouraged.
+
+    Other attributes are taken from Dublin Core terms
+    (https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#section-2)
+    to encourage standardization. This may also be extended to include
+    other attributes.
+
+    """
+
+    # these initial attributes mirror DCMI, and typically apply to an
+    # AlignmentGroup
+    #
+    # which version of the alignment spec this conforms to
+    conformsTo: str = ""
+    # "An entity responsible for making contributions to the
+    # resource". Could be used for alignment annotators. If there are
+    # multiple values, concatenate with commas.
+    contributor: str = ""
+    # strongly recommended if available: "Date of creation of the resource."
+    created: Optional[dt.datetime] = None
+    # strongly recommended if available: "An entity responsible for
+    # making the resource."
+    # Recommended usage: the organization providing the data.
+    creator: str = ""
+    # recommended standard values if defined: "NT", "OT", "DC", a USFM
+    # abbreviation for Bible book
+    coverage: str = ""
+    # "An account of the resource."
+    description: str = ""
+    # for AlignmentRecords: unique identifier
+    id: str = ""
+    # for AlignmentRecords: how was this alignment originally created?
+    # This does *not* capture changes to the original value.
+    # common values include 'manual', 'automated' or an algorithm name
+    origin: str = ""
+    # for ClearAligner to track status. Output here should always be
+    # 'created': set in Manager._make_record() so it's not set of AlignmentGroup metadata
+    # eventually i may need to separate this into two classes, one for
+    # Groups and one for Records.
+    status: str = ""
+    _fieldnames: tuple[str, ...] = ()
+
+    def __post_init__(self) -> None:
+        """Compute values after initialization."""
+        self._fieldnames = tuple(
+            sorted(tuple([f.name for f in fields(self) if f.name != "_fieldnames"]))
+        )
+
+    def __repr__(self) -> str:
+        """Return a printed representation."""
+        attrstr: str = " ".join(
+            {f"{f}={repr(fattr)}" for f in self._fieldnames if (fattr := getattr(self, f))}
+        )
+        return f"Metadata({attrstr})"
+
+    # no hoist option here: the caller decides
+    def asdict(self) -> dict[str, Any]:
+        """Return a dict of values for serialization."""
+        metadict = {f: fattr for f in sorted(self._fieldnames) if (fattr := getattr(self, f))}
+        return metadict
+
+
+@dataclass
+@total_ordering
+class AlignmentRecord:
+    """Manage data for an alignment record."""
+
+    # metadata for this record only
+    meta: Metadata
+    # keys are roles, corresponding to type.roles
+    references: dict[str, AlignmentReference]
+    # TranslationType wires roles to 'source' and 'target'
+    type: TranslationType = field(default_factory=TranslationType)
+
+    def __post_init__(self) -> None:
+        """Compute values after initialization."""
+        for role in self.roles:
+            assert role in self.references, f"role missing from references: {role}"
+        assert len(self.roles) == len(self.references), "different numbers of roles and references"
+
+    def __repr__(self) -> str:
+        """Return a printed representation."""
+        return f"<AlignmentRecord: {repr(self.references)}>"
+
+    def __hash__(self) -> int:
+        """Return a hash value for the record."""
+        return hash(self.identifier)
+
+    def __eq__(self, other):
+        assert isinstance(other, AlignmentRecord), f"Not an AlignmentRecord: {other}"
+        return self.source_selectors[0] == other.source_selectors[0]
+
+    def __lt__(self, other):
+        assert isinstance(other, AlignmentRecord), f"Not an AlignmentRecord: {other}"
+        return self.source_selectors[0] < other.source_selectors[0]
+
+    @property
+    def identifier(self) -> str:
+        """Return the identifier for this record, for convenience."""
+        return self.meta.id
+
+    @property
+    def roles(self) -> tuple[str, str]:
+        """Return the roles for this type, for convenience."""
+        return self.type.roles
+
+    def get_selectors(self, role: str) -> list[str]:
+        """Return the list of selectors for role."""
+        assert role in self.roles, f"Invalid role: {role}"
+        return self.references[role].selectors
+
+    @property
+    def source_selectors(self) -> list[str]:
+        """Return the source selectors for this record."""
+        return self.get_selectors("source")
+
+    @property
+    def target_selectors(self) -> list[str]:
+        """Return the target selectors for this record."""
+        return self.get_selectors("target")
+
+    @property
+    def source_bcv(self) -> str:
+        """Return the source BCV identifier for this record.
+
+        Returns data for the first selector, though multiples should
+        have the same BCV.
+
+        """
+        if self.source_selectors:
+            firstbcv: str = [bcvwpid.to_bcv(sel) for sel in self.source_selectors][0]
+            return firstbcv
+        else:
+            return ""
+
+    @property
+    def incomplete(self) -> bool:
+        """True if any selectors in references are incomplete."""
+        return any(ref.incomplete for ref in self.references.values())
+
+    def asdict(
+        self, positional: bool = False, withmeta: bool = True, withmaculaprefix: bool = False
+    ) -> dict[str, Any]:
+        """Return a dict of values suitable for serialization.
+
+        With positional=False (the default), returns a dict whose keys
+        are the roles and values are the references. Otherwise, the
+        single key is 'references', and the position is determined by
+        the position of the roles.
+
+        With withmeta=False (the default), omits record-level
+        metadata: otherwise includes it.
+
+        With withmaculaprefix=True (the default), prefix source
+        references with 'o' or 'n' depending on canon.
+
+        """
+        recdict: dict[str, Any] = {}
+        if positional:
+            if not withmaculaprefix:
+                raise NotImplementedError(
+                    "Positional and not withmaculaprefix is not yet supported."
+                )
+            else:
+                recdict["references"] = self.references.items()
+        else:
+            # typical case
+            sourcerefs: list[str] = self.references["source"].selectors
+            if withmaculaprefix:
+                # default: add back the Macula prefix
+                sourcerefs = [macula_prefixer(srcstr) for srcstr in sourcerefs]
+            # else leave as is (atypical)
+            recdict["source"] = sourcerefs
+            recdict["target"] = self.references["target"].selectors
+        if withmeta:
+            recdict.update(
+                {
+                    "meta": self.meta.asdict(),
+                }
+            )
+        # sort by keys as Mike prefers
+        return {k: recdict[k] for k in sorted(recdict)}
+
+
+@dataclass
+class AlignmentGroup:
+    """Manage a full set of alignment records.
+
+    This is opinionated about the composition of the group:
+    - enforces a single type across all records
+    """
+
+    # same order and count as roles
+    documents: tuple[Document, Document]
+    # metadata for the group as a whole
+    meta: Metadata
+    records: list[AlignmentRecord]
+    # keys to AlignmentRecord.references: same order as documents
+    roles: tuple[str, str] = ("source", "target")
+    # either "ot" or "nt", based on documents.docid
+    sourcedocid: str = ""
+    canon: str = ""
+    _type: str = ""
+    # hoist docid values from reference.document up to this metadata
+    _hoist_docid: bool = True
+
+    def __post_init__(self) -> None:
+        """Compute values and do checks after initialization."""
+        # only a single type across all records
+        typeset = {rec.type.type for rec in self.records if self.records}
+        assert len(typeset) == 1, f"Multiple AlignmentRecord types found: {typeset}"
+        self._type = typeset.pop()
+        assert len(self.documents) == len(
+            self.roles
+        ), f"Must have same number of documents and roles: {self.documents}, {self.roles}"
+        # one of the documents should have a non-null sourceid: use it
+        # to set the canon for the group
+        sourcedocid = self.documents[0].sourceid or self.documents[1].sourceid
+        assert (
+            sourcedocid
+        ), f"Neither {self.documents[0].docid} nor {self.documents[1].docid} are recognized as source texts:\ncheck src/SourceidEnum for completeness."
+        self.canon = sourcedocid.canon
+        self.sourcedocid = sourcedocid.value
+
+    def __repr__(self) -> str:
+        """Return a printed representation."""
+        docids: tuple[str, str] = tuple([doc.asdict()["docid"] for doc in self.documents])
+        return f"<AlignmentGroup{docids}: {len(self.records)} records>"
+
+    def asdict(self, hoist: bool = True) -> dict[str, Any]:
+        """Return a dict of values suitable for serialization.
+
+        This is opinionated about the preferred serialization: hoists
+        as much as possible to upper levels.
+
+        """
+        # for now
+        positional: bool = False
+        withmeta: bool = False
+
+        return {
+            "meta": self.meta.asdict(),
+            "type": self._type,
+            "records": [
+                rec.asdict(positional=positional, withmeta=withmeta) for rec in self.records
+            ],
+        }
+
+    def verserecords(self) -> dict[str, list[AlignmentRecord]]:
+        """Return a dict mapping source BCV references to their alignment records."""
+        verserecords: dict[str, list[AlignmentRecord]] = {
+            k: list(g) for k, g in groupby(self.records, lambda r: r.source_bcv)
+        }
+        return verserecords
+
+
+# not for Alignments: 2024-09-09
+@dataclass
+class TopLevelGroups:
+    """Manage a pair of AlignmentGroups. Both groups are required."""
+
+    # one group for OT, one group for NT
+    groups: tuple[AlignmentGroup, AlignmentGroup]
+    format: str = "alignment"
+    version: str = "0.3.1"
+    sourcedocids: tuple[str, str] = ()
+    targetdocid: str = ""
+
+    def __post_init__(self) -> None:
+        """Compute values on initialization."""
+        assert len(self.groups) == 2, "There must be two groups."
+        # everything below assumes two groups
+        assert (
+            self.groups[0].roles == self.groups[1].roles
+        ), f"Roles must match: {self.groups[0].roles}, {self.groups[1].roles}"
+        assert (
+            self.groups[0].meta.conformsTo == self.groups[1].meta.conformsTo
+        ), f"meta.conformsto values must match: {self.groups[0].meta.conformsTo}, {self.groups[1].meta.conformsTo}"
+        # target documents should also match
+        targetdocids = list(
+            {group.documents[group.roles.index("target")].docid for group in self.groups}
+        )
+        assert len(targetdocids) == 1, f"OT and NT target docids must match: {targetdocids}"
+        self.targetdocid = targetdocids[0]
+        # canons must be different
+        assert {self.groups[0].canon, self.groups[1].canon} == {
+            "ot",
+            "nt",
+        }, "Both OT and NT canons are required."
+        self.sourcedocids = (self.groups[0].sourcedocid, self.groups[1].sourcedocid)
+
+    def __repr__(self) -> str:
+        """Return a printed representation."""
+        return f"<TopLevelGroups({self.targetdocid}): {self.sourcedocids}>"
+
+    def asdict(self, hoist: bool = True) -> dict[str, Any]:
+        """Return an opionated dict of values suitable for serialization."""
+        return {
+            "format": self.format,
+            "version": self.version,
+            "groups": [self.groups[0].asdict(hoist=hoist), self.groups[1].asdict(hoist=hoist)],
+        }