goldrush 0.0.1__tar.gz

goldrush-0.0.1/LICENSE

@@ -0,0 +1,25 @@
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+this list of conditions and the following disclaimer in the documentation
+and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+Copyright for this project is held by its many contributors, including:
+
+Jane Sandberg <js7389@princeton.edu>
+Ed Summers <ehs@pobox.com>

goldrush-0.0.1/PKG-INFO

@@ -0,0 +1,67 @@
+Metadata-Version: 2.3
+Name: goldrush
+Version: 0.0.1
+Summary: Generate Gold Rush key for pymarc records
+Author: Jane Sandberg, Ed Summers
+Author-email: Jane Sandberg <js7389@princeton.edu>, Ed Summers <ehs@pobox.com>
+License: Redistribution and use in source and binary forms, with or without
+         modification, are permitted provided that the following conditions are met:
+         
+         1. Redistributions of source code must retain the above copyright notice, this
+         list of conditions and the following disclaimer.
+         
+         2. Redistributions in binary form must reproduce the above copyright notice,
+         this list of conditions and the following disclaimer in the documentation
+         and/or other materials provided with the distribution.
+         
+         THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+         ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+         WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+         DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+         FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+         DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+         SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+         CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+         OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+         OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+         
+         Copyright for this project is held by its many contributors, including:
+         
+         Jane Sandberg <js7389@princeton.edu>
+         Ed Summers <ehs@pobox.com>
+Requires-Dist: pymarc>=5.3.1
+Requires-Dist: unidecode>=1.4.0
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+
+# goldrush
+
+**Note: This is alpha software. If you rely on this library you should do so with the
+understanding that you might find errors in the Gold Rush key that is generated.
+Lets make it better together!**
+
+---
+
+[![Build status](https://gitlab.com/pymarc/goldrush/badges/main/pipeline.svg)](https://gitlab.com/pymarc/goldrush/-/commits/main)
+
+*goldrush* is a Python implementation of the [Gold Rush match key algorithm]
+for identifying "duplicate" MARC records, or records that appear to be about
+the same book. It provides a function that you pass a pymarc Record and get
+back the Gold Rush key as a string.
+
+goldrush was largely adapted from [pymarc_dedupe], which included other
+functionality unrelated to Gold Rush, and was not available on PyPI as a
+dependency.
+
+## Usage
+
+```python
+>>> from goldrush import goldrush
+>>> from pymarc import MARCReader
+>>> record = next(MARCReader(open('marc.dat', 'rb')))
+>>> goldrush(record)
+'pragmaticprogrammerfromjourneymantomaster_____________________________2000321____addisa_______________________________________________________hunt________________p'
+```
+
+[Gold Rush match key algorithm]: https://coalliance.org/sites/default/files/GoldRush-Match_KeyJanuary2024_0.doc
+[pymarc_dedupe]: https://github.com/pulibrary/pymarc_dedupe

goldrush-0.0.1/README.md

@@ -0,0 +1,31 @@
+# goldrush
+
+**Note: This is alpha software. If you rely on this library you should do so with the
+understanding that you might find errors in the Gold Rush key that is generated.
+Lets make it better together!**
+
+---
+
+[![Build status](https://gitlab.com/pymarc/goldrush/badges/main/pipeline.svg)](https://gitlab.com/pymarc/goldrush/-/commits/main)
+
+*goldrush* is a Python implementation of the [Gold Rush match key algorithm]
+for identifying "duplicate" MARC records, or records that appear to be about
+the same book. It provides a function that you pass a pymarc Record and get
+back the Gold Rush key as a string.
+
+goldrush was largely adapted from [pymarc_dedupe], which included other
+functionality unrelated to Gold Rush, and was not available on PyPI as a
+dependency.
+
+## Usage
+
+```python
+>>> from goldrush import goldrush
+>>> from pymarc import MARCReader
+>>> record = next(MARCReader(open('marc.dat', 'rb')))
+>>> goldrush(record)
+'pragmaticprogrammerfromjourneymantomaster_____________________________2000321____addisa_______________________________________________________hunt________________p'
+```
+
+[Gold Rush match key algorithm]: https://coalliance.org/sites/default/files/GoldRush-Match_KeyJanuary2024_0.doc
+[pymarc_dedupe]: https://github.com/pulibrary/pymarc_dedupe

goldrush-0.0.1/pyproject.toml

@@ -0,0 +1,26 @@
+[project]
+name = "goldrush"
+version = "0.0.1"
+description = "Generate Gold Rush key for pymarc records"
+readme = "README.md"
+license = {file = "LICENSE"}
+authors = [
+    { name = "Jane Sandberg", email = "js7389@princeton.edu" },
+    { name = "Ed Summers", email = "ehs@pobox.com" }
+]
+requires-python = ">=3.13"
+dependencies = [
+    "pymarc>=5.3.1",
+    "unidecode>=1.4.0",
+]
+
+[build-system]
+requires = ["uv_build>=0.11.6,<0.12.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.3",
+    "ruff>=0.15.11",
+    "ty>=0.0.31",
+]

goldrush-0.0.1/src/goldrush/__init__.py

@@ -0,0 +1,219 @@
+import re
+from functools import cache
+from unidecode import unidecode
+
+from pymarc import Record
+
+from goldrush.record import GoldrushRecord
+
+
+def goldrush(pymarc_rec: Record) -> str:
+    """
+    Pass in a pymarc.Record object and get back a Gold Rush key as a string.
+    """
+    rec = GoldrushRecord(pymarc_rec)
+    return (
+        _title(rec)
+        + (_publication_year(rec))
+        + _pagination(rec)
+        + _edition(rec)
+        + _publisher_name(rec)
+        + rec.type_of()
+        + _title_part(rec)
+        + _title_number(rec)
+        + _title_inclusive_dates(rec)
+        + _author(rec)
+        + _gov_doc_number(rec)
+        + _format_character(rec)
+    )
+
+
+def _title(rec: GoldrushRecord) -> str:
+    some_string = _normalize_title(rec.title())
+    return _pad(some_string, 70)
+
+
+def _title_part(rec: GoldrushRecord) -> str:
+    some_string = _normalize_title(rec.title_part())
+    return _pad(some_string, 30)
+
+
+def _publication_year(rec: GoldrushRecord) -> str:
+    some_string = str(rec.publication_year() or "")
+    return _pad(some_string, 4)
+
+
+def _pagination(rec: GoldrushRecord) -> str:
+    try:
+        nums = re.findall(r"\d+", rec.pagination())[0]
+    except (TypeError, IndexError):
+        nums = ""
+    return _pad(nums, 4)
+
+
+def _edition(rec: GoldrushRecord) -> str:
+    lowercase_ed = ""
+    try:
+        lowercase_ed = rec.edition().lower()
+        chars = re.findall(r"\d+", lowercase_ed)[0]
+    except (TypeError, IndexError):
+        try:
+            chars = re.findall(r"\w+", lowercase_ed)[0]
+            chars = chars[0:3]
+            for ordinal, numeral in _edition_dictionary().items():
+                chars = chars.replace(ordinal, numeral)
+        except (TypeError, IndexError):
+            chars = ""
+    return unidecode(_pad(chars, 3))
+
+
+def _publisher_name(rec: GoldrushRecord) -> str:
+    publisher_name = rec.publisher_name().lower()
+    publisher_name = publisher_name.translate(
+        str.maketrans(_title_translation_dictionary())
+    )
+    return _pad(publisher_name, 5)
+
+
+def _title_number(rec: GoldrushRecord) -> str:
+    title_number = rec.title_number() or ""
+    return _pad(title_number, 10)
+
+
+def _title_inclusive_dates(rec: GoldrushRecord) -> str:
+    title_dates = rec.title_inclusive_dates() or ""
+    title_dates = _strip_punctuation(title_dates)
+    return _pad(title_dates, 15)
+
+
+def _author(rec: GoldrushRecord) -> str:
+    author = rec.author() or ""
+    author = _strip_punctuation(author)
+    return _pad(author, 5)
+
+
+def _gov_doc_number(rec: GoldrushRecord) -> str:
+    gov_doc_number = rec.gov_doc_number() or ""
+    gov_doc_number = _strip_punctuation(gov_doc_number)
+    return _pad(gov_doc_number, 15)
+
+
+def _format_character(rec: GoldrushRecord) -> str:
+    if rec.is_electronic_resource():
+        return "e"
+    return "p"
+
+
+def _normalize_title(some_string: str) -> str:
+    some_string = some_string.strip().lower()
+    # remove beginning English articles
+    some_string = re.sub("^the +", "", some_string)
+    some_string = re.sub("^a +", "", some_string)
+    some_string = re.sub("^an +", "", some_string)
+    some_string = some_string.translate(str.maketrans(_title_translation_dictionary()))
+    return unidecode(some_string)
+
+
+def _pad(string: str, length: int) -> str:
+    return string.ljust(length, "_")[0:length]
+
+
+def _strip_punctuation(some_string: str) -> str:
+    some_string = re.sub("  ", " ", some_string).strip().lower()
+    some_string = re.sub("^the +", "", some_string)
+    some_string = some_string.translate(str.maketrans(_translation_dictionary()))
+    return unidecode(some_string)
+
+
+@cache
+def _edition_dictionary() -> dict[str, str]:
+    return {
+        "fir": "1__",
+        "sec": "2__",
+        "thr": "3__",
+        "fou": "4__",
+        "fiv": "5__",
+        "six": "6__",
+        "sev": "7__",
+        "eig": "8__",
+        "nin": "9__",
+        "ten": "10_",
+    }
+
+
+@cache
+def _title_translation_dictionary() -> dict[str, str]:
+    return {
+        "&": "and",
+        "%": "",
+        "'": "",
+        "{": "",
+        "}": "",
+        " ": "",
+        "!": "",
+        '"': "",
+        "#": "",
+        "$": "",
+        "(": "",
+        ")": "",
+        "*": "",
+        "+": "",
+        ",": "",
+        "-": "",
+        ".": "",
+        "/": "",
+        ":": "",
+        ";": "",
+        "<": "",
+        "=": "",
+        ">": "",
+        "?": "",
+        "@": "",
+        "[": "",
+        "\\": "",
+        "]": "",
+        "^": "",
+        "`": "",
+        "|": "",
+        "~": "",
+        "©": "",
+    }
+
+
+@cache
+def _translation_dictionary() -> dict[str, str]:
+    return {
+        "&": "and",
+        "%": "_",
+        "'": "",
+        "{": "",
+        "}": "",
+        " ": "_",
+        "!": "_",
+        '"': "_",
+        "#": "_",
+        "$": "_",
+        "(": "_",
+        ")": "_",
+        "*": "_",
+        "+": "_",
+        ",": "_",
+        "-": "_",
+        ".": "_",
+        "/": "_",
+        ":": "_",
+        ";": "_",
+        "<": "_",
+        "=": "_",
+        ">": "_",
+        "?": "_",
+        "@": "_",
+        "[": "_",
+        "\\": "_",
+        "]": "_",
+        "^": "_",
+        "`": "_",
+        "|": "_",
+        "~": "_",
+        "©": "_",
+    }

goldrush-0.0.1/src/goldrush/record.py

@@ -0,0 +1,292 @@
+import re
+import string
+
+import pymarc
+
+
+class GoldrushRecord:
+    def __init__(self, record):
+        self.record = record
+
+    def id(self):
+        try:
+            return self.record.get("001").data
+        except (KeyError, AttributeError):
+            return ""
+
+    def title(self):
+        if self.__vernacular_title_field():
+            title_field = self.__vernacular_title_field()
+        else:
+            title_field = self.__title_from_245()
+        try:
+            subfield_a = str(title_field.get("a") or "")
+            subfield_b = str(title_field.get("b") or "")
+            subfield_p = str(title_field.get("p") or "")
+            title = " ".join([subfield_a, subfield_b, subfield_p])
+            title = self.__strip_ending_punctuation(title)
+            return title
+        except (KeyError, AttributeError):
+            return ""
+
+    def transliterated_title(self):
+        title_field = self.__title_from_245()
+        try:
+            subfield_a = str(title_field.get("a") or "")
+            subfield_b = str(title_field.get("b") or "")
+            subfield_p = str(title_field.get("p") or "")
+            title = " ".join([subfield_a, subfield_b, subfield_p])
+            title = self.__strip_ending_punctuation(title)
+            return title
+        except (KeyError, AttributeError):
+            return ""
+
+    def __title_from_245(self):
+        try:
+            title = self.record["245"]
+            return title
+        except KeyError:
+            return ""
+
+    def __vernacular_title_field(self):
+        try:
+            return self.record.get_linked_fields(self.record["245"])[0]
+        except (KeyError, IndexError, pymarc.exceptions.MissingLinkedFields):
+            return ""
+
+    def publication_year(self):
+        pub_year = None
+        if self.date_one() and self.date_two():
+            pub_year = self.date_two()
+        elif self.date_one() and not self.date_two():
+            pub_year = self.date_one()
+        elif (
+            not self.date_one() and not self.date_two() and self.__date_of_production()
+        ):
+            pub_year = self.__date_of_production()
+        elif (
+            not self.date_one()
+            and not self.date_two()
+            and not self.__date_of_production()
+            and self.__date_of_publication()
+        ):
+            pub_year = self.__date_of_publication()
+        return pub_year
+
+    def pagination(self):
+        try:
+            subfield_a = self.record["300"].get("a")
+            if subfield_a:
+                return self.__normalize_extent(subfield_a)
+            return ""
+        except KeyError:
+            return ""
+
+    def edition(self):
+        try:
+            return self.__normalize_edition(self.record["250"].get("a"))
+        except KeyError:
+            return ""
+
+    def publisher_name(self):
+        try:
+            pub = self.record["264"]["b"]
+        except KeyError:
+            try:
+                pub = self.record["260"]["b"]
+            except KeyError:
+                return ""
+        return self.__strip_punctuation(pub)
+
+    def type_of(self):
+        return self.record.leader.type_of_record
+
+    def title_part(self):
+        try:
+            parts = self.record["245"].get_subfields("p")[1:]
+            return self.__strip_punctuation(" ".join(parts))
+        except KeyError:
+            return ""
+
+    def title_number(self):
+        try:
+            num = self.record["245"].get("n")
+            if num:
+                return self.__strip_punctuation(num)
+            return ""
+        except KeyError:
+            return ""
+
+    def author(self):
+        if self.__vernacular_author_field():
+            author_field = self.__vernacular_author_field()
+        else:
+            author_field = self.__author_from_1xx()
+
+        if author_field:
+            try:
+                return self.__strip_ending_punctuation(author_field.get("a"))
+            except AttributeError:
+                return ""
+        return ""
+
+    def __author_from_1xx(self):
+        try:
+            return self.record["100"]
+        except KeyError:
+            try:
+                return self.record["110"]
+            except KeyError:
+                try:
+                    return self.record["111"]
+                except KeyError:
+                    return ""
+
+    def __vernacular_author_field(self):
+        try:
+            return self.record.get_linked_fields(self.record["100"])[0]
+        except (KeyError, IndexError, pymarc.exceptions.MissingLinkedFields):
+            try:
+                return self.record.get_linked_fields(self.record["110"])[0]
+            except (KeyError, IndexError, pymarc.exceptions.MissingLinkedFields):
+                try:
+                    return self.record.get_linked_fields(self.record["111"])[0]
+                except (KeyError, IndexError, pymarc.exceptions.MissingLinkedFields):
+                    return ""
+
+    def title_inclusive_dates(self):
+        try:
+            date = self.record["245"].get("f")
+            if date:
+                return self.__strip_ending_punctuation(date)
+            return ""
+        except KeyError:
+            return ""
+
+    def gov_doc_number(self):
+        try:
+            return self.record["086"].get("a")
+        except KeyError:
+            return ""
+
+    def is_electronic_resource(self):
+        return bool(
+            self.__is_electronic_resource_from_title()
+            or self.__is_electronic_resource_from_reproduction()
+            or self.__is_electronic_resource_from_description()
+            or self.__is_electronic_resource_from_007()
+        )
+
+    def __is_electronic_resource_from_title(self):
+        try:
+            return self.record["245"].get("h") == "[electronic resource]"
+        except KeyError:
+            return False
+
+    def __is_electronic_resource_from_reproduction(self):
+        try:
+            return re.match(
+                "electronic reproduction", self.record["533"].get("a"), re.IGNORECASE
+            )
+        except (KeyError, TypeError):
+            return False
+
+    def __is_electronic_resource_from_description(self):
+        try:
+            subfield_a = self.record["300"].get("a")
+            if subfield_a:
+                return bool(re.search("online resource", subfield_a, re.IGNORECASE))
+            return False
+        except KeyError:
+            return False
+
+    def __is_electronic_resource_from_007(self):
+        try:
+            return bool(self.record["007"].data[0] == "c")
+        except KeyError:
+            return False
+
+    def __normalize_edition(self, edition):
+        edition_mapping = {r"Ed\.": "Edition", r"ed\.": "edition"}
+        try:
+            for key, value in edition_mapping.items():
+                edition = re.sub(key, value, edition)
+            return self.__strip_punctuation(edition)
+        except TypeError:
+            return ""
+
+    def __normalize_extent(self, extent):
+        extent_mapping = {
+            r"p\.": "pages",
+            r"v\.": "volumes",
+            r"vol\.": "volumes",
+            r"ℓ\.": "leaves",
+        }
+        for key, value in extent_mapping.items():
+            extent = re.sub(key, value, extent)
+        return self.__strip_punctuation(extent)
+
+    def __strip_ending_punctuation(self, some_string):
+        punctuation_to_strip = string.punctuation.replace(")", "")
+        return some_string.strip(punctuation_to_strip + " ")
+
+    def __strip_punctuation(self, some_string):
+        punctuation_to_strip = string.punctuation.replace("&", "")
+        some_string = some_string.translate(str.maketrans("", "", punctuation_to_strip))
+        some_string = re.sub("  ", " ", some_string).strip()
+        return some_string
+
+    def is_valid_date(self, date_string):
+        valid = True
+        if date_string == "9999":
+            valid = False
+        elif date_string == "    ":
+            valid = False
+        elif self.number_of_characters(date_string) != 4:
+            valid = False
+        try:
+            int(date_string)
+        except (ValueError, TypeError):
+            valid = False
+        return valid
+
+    def number_of_characters(self, date_string):
+        try:
+            return len(date_string)
+        except TypeError:
+            return False
+
+    def date_one(self):
+        try:
+            date_string = self.record["008"].data[7:11]
+            return self.__as_date(date_string)
+        except KeyError:
+            return None
+
+    def date_two(self):
+        try:
+            date_string = self.record["008"].data[11:15]
+            return self.__as_date(date_string)
+        except KeyError:
+            return None
+
+    def __date_of_production(self):
+        try:
+            date_string = self.record["264"]["c"]
+        except KeyError:
+            return None
+        return self.__as_date(date_string)
+
+    def __date_of_publication(self):
+        try:
+            date_string = self.record["260"]["c"]
+        except KeyError:
+            return ""
+        return self.__as_date(date_string)
+
+    def __as_date(self, date_string):
+        # Remove punctuation (for 260 and 264 fields)
+        date_string = self.__strip_punctuation(date_string)
+        if self.is_valid_date(date_string):
+            return int(date_string)
+        return ""