marcgrep 0.2.0__tar.gz

marcgrep-0.2.0/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.1
+Name: marcgrep
+Version: 0.2.0
+Summary: search MARC files for regex matches
+Home-page: https://github.com/phette23/marcgreppy
+License: MIT
+Keywords: marc,grep,regex,libraries,cli,metadata,bibliographic,cataloging
+Author: phette23
+Author-email: phette23@gmail.com
+Requires-Python: >3.8
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Utilities
+Requires-Dist: click (==8.1.7)
+Requires-Dist: pymarc (==5.1.2)
+Project-URL: Issues, https://github.com/phette23/marcgreppy/issues
+Project-URL: Repository, https://github.com/phette23/marcgreppy
+Description-Content-Type: text/markdown
+
+# marcgrep
+
+A CLI for searching MARC files like [MARCgrep.pl](https://pusc.it/bib/MARCgrep) but in Python and a bit different syntax.
+
+[marcli](https://github.com/hectorcorrea/marcli) is also a similar project that's faster but a little less flexible.
+
+## Installation
+
+Python 3.9 or later. [Poetry](https://python-poetry.org/) is used for development.
+
+```sh
+pipx install marcgrep # install globally with pipx
+pip install marcgrep # or use pip/pip3
+```
+
+## Usage
+
+```sh
+# general command format
+$ marcgrep OPTIONS FILE.mrc
+$ cat FILE.mrc | marcgrep OPTIONS
+# full usage information
+$ marcgrep -h
+Usage: marcgrep [OPTIONS] [FILE]
+
+  Find MARC records matching patterns in a file.
+
+Options:
+  -h, --help          Show this message and exit.
+  -v, --version       Show marcgrep version
+  -c, --count         Count matching records
+  -i, --include TEXT  Include matching records (repeatable)
+  -e, --exclude TEXT  Exclude matching records (repeatable)
+```
+
+The `--include` and `--exclude` flags can be used multiple times to specify multiple criteria. They accept a pattern which is a sort of comma-separated filter expression for matching MARC fields. Examples:
+
+```sh
+# records with a 780 field
+$ marcgrep -i 780 FILE.mrc
+# records with Ulysses in the 245 field
+$ marcgrep -i '245,Ulysses' FILE.mrc
+# titles _without_ "Collected Poems" in the 245 $a subfield
+$ marcgrep -e '245,a,Collected Poems' FILE.mrc
+# titles with second indicator = 4 that do not start with "The "
+$ marcgrep -i '245,,4,,^(?!The )' FILE.mrc
+```
+
+The meaning of the pattern's components depends upon their number:
+
+- 1: field, `910` -> 910 is in record
+- 2: field and value (regular expression), `100,Lorde` -> 100 contains string "Lorde"
+- 3: field, subfield, and value, `506,a,Open Access` -> 506$a contains string "Open Access"
+- 4: field, subfield, first indicator, and value, `856,0,u,@lcsh\.gov` -> 856$u with 1st indicator 0 contains string "@lcsh.gov"
+- 5: field, subfield, first & second indicators, and value, `245,0,4,a,The Communist Manifesto`
+
+The intention of this syntax is to facilitate searching subfields and field values more easily than MARCgrep.pl since we care about them more often than indicators. To ignore a component but use one of lesser priority, leave the component empty. For instance, `856,s,` refers to records with an `856` field with a `$s` subfield but the trailing comma means we don't care about the subfield's value. The pattern `245,,4,,` refers to records with a `245` field with a second indicator of `4` regardless its subfields or value.
+
+Multiple criteria are combined with logical AND. Multiple `--include` flags is narrower than one, as is an `--include` and an `--exclude`.
+
+## Development
+
+- [x] -c count
+- [x] -v version
+- [ ] -l limit (number of records to process)
+- [x] -i include criteria (multiple)
+- [x] -e exclude criteria (multiple)
+- [ ] -f fields to print
+- [ ] work with MARC leader
+- [ ] regex for all components? e.g. `24.,text in any 240-249 field`
+- [ ] relatedly, specify _not_ to treat value as a regex?
+
+```sh
+poetry install # install dependencies
+poetry run pytest # run tests
+```
+
+## License
+
+[MIT](https://opensource.org/license/mit) © Eric Phetteplace 2024.
+

marcgrep-0.2.0/marcgrep/cli.py

@@ -0,0 +1,67 @@
+from importlib import metadata
+from typing import BinaryIO
+
+import click
+from pymarc import MARCReader
+
+from .filter import Filter
+from .utils import count_records
+
+
+@click.command(help="Find MARC records matching patterns in a file.")
+@click.help_option("-h", "--help")
+# TODO explain pattern syntax in command line help
+@click.argument("file", type=click.File("rb"), default="-")
+@click.option("--version", "-v", help="Show marcgrep version", is_flag=True)
+@click.option("--count", "-c", help="Count matching records", is_flag=True)
+@click.option(
+    "--include", "-i", help="Include matching records (repeatable)", multiple=True
+)
+@click.option(
+    "--exclude", "-e", help="Exclude matching records (repeatable)", multiple=True
+)
+def main(
+    file: BinaryIO,
+    count: bool,
+    version: bool,
+    include: list[str],
+    exclude: list[str],
+):
+    if version:
+        print(metadata.version("marcgrep"))
+        return exit(0)
+
+    num_records = 0
+    reader = MARCReader(file)
+
+    # build a list of filters, start with exclusive because they rule out records quicker
+    filters = [Filter(pattern, inclusive=False) for pattern in exclude]
+    filters.extend(Filter(pattern) for pattern in include)
+
+    # if no filters, count or print all records
+    if not len(filters):
+        if count:
+            num_records = count_records(reader)
+            print(num_records)
+            # non-zero exit if no records found
+            return exit(0 if num_records else 1)
+        for record in reader:
+            print(record)
+        return exit(0)
+
+    for record in reader:
+        if record:
+            if all(f.match(record) for f in filters):
+                num_records += 1
+                if not count:
+                    print(record)
+
+    if count:
+        print(num_records)
+
+    # non-zero exit if no records match
+    return exit(0 if num_records else 1)
+
+
+if __name__ == "__main__":
+    main()

marcgrep-0.2.0/marcgrep/filter.py

@@ -0,0 +1,94 @@
+import re
+
+from pymarc import Record
+
+
+def parse_pattern(pattern):
+    parts = pattern.split(",")
+    # Convert empty strings to None to support "123,,,,value" use case
+    parts = [part if part else None for part in parts]
+    # 3.10 Match-Case would be better here
+    # We prioritize the field, then the value, then the subfield, and indicators come last
+    if len(parts) == 1:
+        return parts[0], None, None, None, None
+    if len(parts) == 2:
+        return parts[0], None, None, None, parts[1]
+    if len(parts) == 3:
+        return parts[0], None, None, parts[1], parts[2]
+    if len(parts) == 4:
+        return parts[0], parts[1], None, parts[2], parts[3]
+    # combine the rest into the value, convert Nones past the 4th index back to empty string
+    value = ",".join([p if p else "" for p in parts[4:]])
+    return parts[0], parts[1], parts[2], parts[3], value
+
+
+class FilterFormatError(Exception):
+    pass
+
+
+class Filter:
+    def __init__(self, pattern, inclusive=True) -> None:
+        """Pettern is a comma-separated string like
+        FIELD,INDICATOR1,INDICATOR2,SUBFIELD,VALUE
+        where only FIELD is required and VALUE can itself contain commas
+        e.g. 245,cat in the hat -> field:245, value:cat in the hat
+        245,4,0,a,cat in the hat -> field:245, ind1:4, ind2:0, subfield:a, value:cat in the hat
+        """
+        self.inclusive = inclusive
+        self.pattern = pattern
+        parts = parse_pattern(pattern)
+        self.field: str | None = parts[0]
+        self.ind1: str | None = parts[1]
+        self.ind2: str | None = parts[2]
+        self.subfield: str | None = parts[3]
+        self.value: str | None = parts[4]
+        self.validate()
+
+    def __repr__(self) -> str:
+        return f"{'Inclusive' if self.inclusive else 'Exclusive'} Filter: field={self.field}, ind1={self.ind1}, ind2={self.ind2}, subfield={self.subfield}, value={self.value}"
+
+    def validate(self) -> None:
+        if not self.field:
+            raise FilterFormatError(
+                "Field is required; filter pattern cannot be empty."
+            )
+        if len(self.field) != 3 or not self.field.isdigit():
+            raise FilterFormatError(
+                f"Field {self.field} is invalid, must be three digits 000-999."
+            )
+        if self.ind1 is not None and (len(self.ind1) != 1 or not self.ind1.isdigit()):
+            raise FilterFormatError(
+                f"Indicator 1 {self.ind1} is invalid, must one one digit 0-9."
+            )
+        if self.ind2 is not None and (len(self.ind2) != 1 or not self.ind2.isdigit()):
+            raise FilterFormatError(
+                f"Indicator 2 {self.ind2} is invalid, must one one digit 0-9."
+            )
+        if self.subfield is not None and len(self.subfield) != 1:
+            raise FilterFormatError(
+                f"Subfield {self.subfield} is invalid, must be one character."
+            )
+
+    def match(self, record: Record) -> bool:
+        result = self.inclusive
+        for field in record.get_fields(self.field):
+            # print(re.match(self.value, field.format_field()))
+            if self.ind1 and field.indicator1 != self.ind1:
+                continue
+            if self.ind2 and field.indicator2 != self.ind2:
+                continue
+            if self.subfield:
+                subfields = field.get_subfields(self.subfield)
+                if (
+                    self.value
+                    and not any(
+                        re.search(self.value, subfield) for subfield in subfields
+                    )
+                    or not len(subfields)
+                ):
+                    continue
+            if self.value and not re.search(self.value, field.format_field()):
+                continue
+            # if we get here, the filter matched
+            return result
+        return not result

marcgrep-0.2.0/marcgrep/utils.py

@@ -0,0 +1,5 @@
+from pymarc import Record
+
+
+def count_records(reader):
+    return sum(1 for _ in reader if isinstance(_, Record))

marcgrep-0.2.0/pyproject.toml

@@ -0,0 +1,39 @@
+[tool.poetry]
+name = "marcgrep"
+version = "0.2.0"
+description = "search MARC files for regex matches"
+authors = ["phette23 <phette23@gmail.com>"]
+license = "MIT"
+readme = "readme.md"
+homepage = "https://github.com/phette23/marcgreppy"
+repository = "https://github.com/phette23/marcgreppy"
+keywords = ["marc", "grep", "regex", "libraries", "cli", "metadata", "bibliographic", "cataloging"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+    "Topic :: Utilities",
+]
+
+[tool.poetry.urls]
+"Issues" = "https://github.com/phette23/marcgreppy/issues"
+
+[tool.poetry.dependencies]
+python = ">3.8"
+pymarc = "5.1.2"
+click = "8.1.7"
+
+[tool.poetry.group.dev.dependencies]
+pytest = "8.1.1"
+
+[tool.poetry.scripts]
+marcgrep = "marcgrep.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["marcgrep"]
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

marcgrep-0.2.0/readme.md

@@ -0,0 +1,80 @@
+# marcgrep
+
+A CLI for searching MARC files like [MARCgrep.pl](https://pusc.it/bib/MARCgrep) but in Python and a bit different syntax.
+
+[marcli](https://github.com/hectorcorrea/marcli) is also a similar project that's faster but a little less flexible.
+
+## Installation
+
+Python 3.9 or later. [Poetry](https://python-poetry.org/) is used for development.
+
+```sh
+pipx install marcgrep # install globally with pipx
+pip install marcgrep # or use pip/pip3
+```
+
+## Usage
+
+```sh
+# general command format
+$ marcgrep OPTIONS FILE.mrc
+$ cat FILE.mrc | marcgrep OPTIONS
+# full usage information
+$ marcgrep -h
+Usage: marcgrep [OPTIONS] [FILE]
+
+  Find MARC records matching patterns in a file.
+
+Options:
+  -h, --help          Show this message and exit.
+  -v, --version       Show marcgrep version
+  -c, --count         Count matching records
+  -i, --include TEXT  Include matching records (repeatable)
+  -e, --exclude TEXT  Exclude matching records (repeatable)
+```
+
+The `--include` and `--exclude` flags can be used multiple times to specify multiple criteria. They accept a pattern which is a sort of comma-separated filter expression for matching MARC fields. Examples:
+
+```sh
+# records with a 780 field
+$ marcgrep -i 780 FILE.mrc
+# records with Ulysses in the 245 field
+$ marcgrep -i '245,Ulysses' FILE.mrc
+# titles _without_ "Collected Poems" in the 245 $a subfield
+$ marcgrep -e '245,a,Collected Poems' FILE.mrc
+# titles with second indicator = 4 that do not start with "The "
+$ marcgrep -i '245,,4,,^(?!The )' FILE.mrc
+```
+
+The meaning of the pattern's components depends upon their number:
+
+- 1: field, `910` -> 910 is in record
+- 2: field and value (regular expression), `100,Lorde` -> 100 contains string "Lorde"
+- 3: field, subfield, and value, `506,a,Open Access` -> 506$a contains string "Open Access"
+- 4: field, subfield, first indicator, and value, `856,0,u,@lcsh\.gov` -> 856$u with 1st indicator 0 contains string "@lcsh.gov"
+- 5: field, subfield, first & second indicators, and value, `245,0,4,a,The Communist Manifesto`
+
+The intention of this syntax is to facilitate searching subfields and field values more easily than MARCgrep.pl since we care about them more often than indicators. To ignore a component but use one of lesser priority, leave the component empty. For instance, `856,s,` refers to records with an `856` field with a `$s` subfield but the trailing comma means we don't care about the subfield's value. The pattern `245,,4,,` refers to records with a `245` field with a second indicator of `4` regardless its subfields or value.
+
+Multiple criteria are combined with logical AND. Multiple `--include` flags is narrower than one, as is an `--include` and an `--exclude`.
+
+## Development
+
+- [x] -c count
+- [x] -v version
+- [ ] -l limit (number of records to process)
+- [x] -i include criteria (multiple)
+- [x] -e exclude criteria (multiple)
+- [ ] -f fields to print
+- [ ] work with MARC leader
+- [ ] regex for all components? e.g. `24.,text in any 240-249 field`
+- [ ] relatedly, specify _not_ to treat value as a regex?
+
+```sh
+poetry install # install dependencies
+poetry run pytest # run tests
+```
+
+## License
+
+[MIT](https://opensource.org/license/mit) © Eric Phetteplace 2024.