ocr-util 2.2.1__tar.gz → 3.0.1__tar.gz

{ocr_util-2.2.1/src/ocr_util.egg-info → ocr_util-3.0.1}/PKG-INFO

@@ -1,11 +1,11 @@
 Metadata-Version: 2.4
 Name: ocr-util
-Version: 2.2.1
+Version: 3.0.1
 Summary: OCR Utils
 Author-email: Universitäts- und Landesbibliothek Sachsen-Anhalt <development@bibliothek.uni-halle.de>
+License-Expression: MIT
 Project-URL: Homepage, https://github.com/ulb-sachsen-anhalt/ocr-util
 Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -38,7 +38,6 @@ Dynamic: license-file
 
 Collection of utils for 
 * evaluation of OCR data for the masses
-* generation of extended OCR-Evaluation Corpora
 * generation of pair-wise Trainingdata for OCR-Backends
 
 ## Requirements
@@ -54,16 +53,10 @@ Each section contains detailed usage help instructions:
 # evaluation
 ocr-util eval --help
 
-# corpus management
-ocr-util corpus --help
-
-# analyse an existing METS corpus
-ocr-util corpus-analyse --help
-
 # slice image by image + input OCR
 ocr-util slice --help
 
-# render image + input OCR
+# render input OCR (regions, lines, words) on given image
 ocr-util show --help
 ```
 
@@ -73,7 +66,7 @@ Inconsistent OCR Groundtruth with empty texts (ALTO String elements missing CONT
 
 _Please note_:  
 Invalid data files are excluded and reported where possible from evaluation.  
-The term 'invalid' refers to errors in schemas in structured XML-data, i.e. syntax errors and further if included geometrical information includes inconsistencies like missing points or missmatching shapes.
+The term 'invalid' refers to malformed or structurally invalid XML, i.e. syntax errors and further if included geometrical information includes inconsistencies like missing points or missmatching shapes.
 
 ### Evaluation Filter-Then-Aggregate
 
@@ -89,6 +82,22 @@ ocr-util eval <candidates> \
 	--aggregate-by "mods:dateIssued:century"
 ```
 
+The dimensions can also be reversed: filter by publication century or decade,
+then aggregate by language. For example, keep only publications from the 17th
+century (1601-1700 inclusive), then group their evaluation results by MODS language:
+
+```bash
+ocr-util eval <candidates> \
+	--reference <groundtruth> \
+	--mets-file <mets.xml> \
+	--filter-by "mods:dateIssued:century=17th" \
+	--aggregate-by "mods:language"
+```
+
+Filtering still happens before evaluation and aggregation; only the metadata
+dimensions exchange roles. To select a decade instead, replace the filter above
+with `--filter-by "mods:dateIssued:decade=1800s"` (1800-1809 inclusive).
+
 Multi-language filter values are interpreted as sets:
 
 ```bash
@@ -100,6 +109,8 @@ ocr-util eval <candidates> \
 ```
 
 Behavior:
+* centuries follow calendar numbering: the 18th century is 1701-1800,
+	and the 19th century is 1801-1900; decades retain ten-year buckets such as 1800-1809
 * single filter value -> exact match (e.g. `ger` does not match `ger+eng`)
 * multi-value filter -> all filter values must be present in any order
 * entries missing the filter criterion are reported as WARNING and discarded
@@ -108,34 +119,6 @@ Behavior:
 * aggregation coverage reports how many evaluated pairs provide each dimension;
 	reported `items` count candidate/ground-truth pairs, not MODS elements
 
-### Corpus Analysis
-
-Existing METS corpora can be filtered without evaluation candidate data. Matching
-full-text file references are printed one per line:
-
-```bash
-ocr-util corpus-analyse <mets.xml> \
-	--filter-by "mods:dateIssued:century=16th"
-```
-
-Repeat `--filter-by` to combine criteria with AND:
-
-```bash
-ocr-util corpus-analyse <mets.xml> \
-	--filter-by "mods:dateIssued:century=16th" \
-	--filter-by "mods:language=ger"
-```
-
-Check that every local file referenced by any METS `FLocat` exists:
-
-```bash
-ocr-util corpus-analyse <mets.xml> --check
-```
-
-Relative paths are resolved from the METS directory. Missing files and remote
-references that cannot be checked locally produce a non-zero exit status. This
-is a file-presence check; it does not perform XML schema validation.
-
 ## Development
 
 Platform: Intel(R) Core(TM) i5-6500 CPU@3.20GHz, 16GB RAM, Ubuntu 22.04 LTS, Python 3.10+

{ocr_util-2.2.1 → ocr_util-3.0.1}/README.md

@@ -5,7 +5,6 @@
 
 Collection of utils for 
 * evaluation of OCR data for the masses
-* generation of extended OCR-Evaluation Corpora
 * generation of pair-wise Trainingdata for OCR-Backends
 
 ## Requirements
@@ -21,16 +20,10 @@ Each section contains detailed usage help instructions:
 # evaluation
 ocr-util eval --help
 
-# corpus management
-ocr-util corpus --help
-
-# analyse an existing METS corpus
-ocr-util corpus-analyse --help
-
 # slice image by image + input OCR
 ocr-util slice --help
 
-# render image + input OCR
+# render input OCR (regions, lines, words) on given image
 ocr-util show --help
 ```
 
@@ -40,7 +33,7 @@ Inconsistent OCR Groundtruth with empty texts (ALTO String elements missing CONT
 
 _Please note_:  
 Invalid data files are excluded and reported where possible from evaluation.  
-The term 'invalid' refers to errors in schemas in structured XML-data, i.e. syntax errors and further if included geometrical information includes inconsistencies like missing points or missmatching shapes.
+The term 'invalid' refers to malformed or structurally invalid XML, i.e. syntax errors and further if included geometrical information includes inconsistencies like missing points or missmatching shapes.
 
 ### Evaluation Filter-Then-Aggregate
 
@@ -56,6 +49,22 @@ ocr-util eval <candidates> \
 	--aggregate-by "mods:dateIssued:century"
 ```
 
+The dimensions can also be reversed: filter by publication century or decade,
+then aggregate by language. For example, keep only publications from the 17th
+century (1601-1700 inclusive), then group their evaluation results by MODS language:
+
+```bash
+ocr-util eval <candidates> \
+	--reference <groundtruth> \
+	--mets-file <mets.xml> \
+	--filter-by "mods:dateIssued:century=17th" \
+	--aggregate-by "mods:language"
+```
+
+Filtering still happens before evaluation and aggregation; only the metadata
+dimensions exchange roles. To select a decade instead, replace the filter above
+with `--filter-by "mods:dateIssued:decade=1800s"` (1800-1809 inclusive).
+
 Multi-language filter values are interpreted as sets:
 
 ```bash
@@ -67,6 +76,8 @@ ocr-util eval <candidates> \
 ```
 
 Behavior:
+* centuries follow calendar numbering: the 18th century is 1701-1800,
+	and the 19th century is 1801-1900; decades retain ten-year buckets such as 1800-1809
 * single filter value -> exact match (e.g. `ger` does not match `ger+eng`)
 * multi-value filter -> all filter values must be present in any order
 * entries missing the filter criterion are reported as WARNING and discarded
@@ -75,34 +86,6 @@ Behavior:
 * aggregation coverage reports how many evaluated pairs provide each dimension;
 	reported `items` count candidate/ground-truth pairs, not MODS elements
 
-### Corpus Analysis
-
-Existing METS corpora can be filtered without evaluation candidate data. Matching
-full-text file references are printed one per line:
-
-```bash
-ocr-util corpus-analyse <mets.xml> \
-	--filter-by "mods:dateIssued:century=16th"
-```
-
-Repeat `--filter-by` to combine criteria with AND:
-
-```bash
-ocr-util corpus-analyse <mets.xml> \
-	--filter-by "mods:dateIssued:century=16th" \
-	--filter-by "mods:language=ger"
-```
-
-Check that every local file referenced by any METS `FLocat` exists:
-
-```bash
-ocr-util corpus-analyse <mets.xml> --check
-```
-
-Relative paths are resolved from the METS directory. Missing files and remote
-references that cannot be checked locally produce a non-zero exit status. This
-is a file-presence check; it does not perform XML schema validation.
-
 ## Development
 
 Platform: Intel(R) Core(TM) i5-6500 CPU@3.20GHz, 16GB RAM, Ubuntu 22.04 LTS, Python 3.10+

{ocr_util-2.2.1 → ocr_util-3.0.1}/pyproject.toml

@@ -7,8 +7,8 @@ requires-python = ">=3.10"
 authors = [{name = "Universitäts- und Landesbibliothek Sachsen-Anhalt",email = "development@bibliothek.uni-halle.de"}]
 classifiers = [
     "Programming Language :: Python :: 3",
-	"License :: OSI Approved :: MIT License"
 ]
+license = "MIT"
 dependencies = [
 	"rapidfuzz>3",
     "nltk",
@@ -53,7 +53,7 @@ version = {attr = "ocr_util.__version__"}
 where = ["src"]
 
 [tool.setuptools.package-data]
-ocr_util = ["corpus/*.xml"]
+ocr_corpus = ["*.xml"]
 
 [tool.setuptools]
 package-dir = {"" = "src"}

ocr_util-3.0.1/src/ocr_util/__init__.py

@@ -0,0 +1,3 @@
+"""main API"""
+
+__version__ = "3.0.1"

{ocr_util-2.2.1 → ocr_util-3.0.1}/src/ocr_util/cli.py

@@ -3,9 +3,8 @@
 
 import argparse
 import logging
-import os
 import re
-from pathlib import Path, PurePath
+from pathlib import PurePath
 
 import ocr_util
 import ocr_util.eval.model as do
@@ -13,18 +12,10 @@ import ocr_util.eval.model.filter as dofi
 import ocr_util.eval.cli as eval_cli
 import ocr_util.slice.cli as slice_cli
 import ocr_util.show.cli as show_cli
-import ocr_util.corpus.analyse as corpus_analyse
-import ocr_util.corpus.generate_corpus as gc
-
-from ocr_util.corpus.common import CorpusArgs
 
 # script constants
 DEFAULT_VERBOSITY = 0
 SUB_CMD_FRAME = "frame"
-SUB_CMD_GROUNDTRUTH_CORPUS = "corpus"
-SUB_CMD_CORPUS_ANALYSE = "corpus-analyse"
-CORPUS_CACHE_DIR_NAME = "ocr_util_corpus_mets_cache"
-CORPUS_CACHE_DIR = os.path.join(os.path.expanduser("~"), ".cache", CORPUS_CACHE_DIR_NAME)
 
 SUB_CMD_EVALUATE = "eval"
 SUB_CMD_SLICE = "slice"
@@ -89,77 +80,6 @@ def start() -> None:
         """,
     )
 
-    # groundtruth-corpus subcommand
-    groundtruth_corpus_arg_parser = sub_arg_parsers.add_parser(
-        SUB_CMD_GROUNDTRUTH_CORPUS,
-        help="Create METS file from N ground truth PAGE-XML files with URN identifiers",
-    )
-    groundtruth_corpus_arg_parser.add_argument(
-        "-i",
-        "--input",
-        dest="input_dir",
-        help="Path to the input directory containing GT PAGE-XML files",
-        required=True,
-    )
-    groundtruth_corpus_arg_parser.add_argument(
-        "-o",
-        "--output",
-        dest="output_dir",
-        help="Path to the output directory for generated corpus",
-        required=True,
-    )
-    groundtruth_corpus_arg_parser.add_argument(
-        "-l",
-        "--limit",
-        type=int,
-        default=0,
-        help="Number of files to process (default: 0 = unlimited)",
-        required=False,
-    )
-    groundtruth_corpus_arg_parser.add_argument(
-        "-t",
-        "--temp-dir",
-        dest="temp_dir",
-        default=CORPUS_CACHE_DIR,
-        help=f"Path to temporary directory for caching METS files (default: {CORPUS_CACHE_DIR})",
-        required=False,
-    )
-    groundtruth_corpus_arg_parser.add_argument(
-        "-v",
-        "--verbosity",
-        action="count",
-        default=DEFAULT_VERBOSITY,
-        required=False,
-        help=f"Verbosity flag. To increase, append multiple 'v's (optional; default: '{DEFAULT_VERBOSITY}')",
-    )
-    groundtruth_corpus_arg_parser.add_argument(
-        "--corpus-label",
-        dest="corpus_label",
-        default="Ground Truth Corpus",
-        help="Label for the corpus in the METS logical structure (default: 'Ground Truth Corpus')",
-        required=False,
-    )
-    groundtruth_corpus_arg_parser.add_argument(
-        "--oai-base-url",
-        dest="oai_base_url",
-        help="Base URL for OAI-PMH requests",
-        required=False,
-    )
-    groundtruth_corpus_arg_parser.add_argument(
-        "--clear-cache",
-        dest="clear_cache",
-        action="store_true",
-        default=False,
-        help="Clear the local cache directory before processing (default: False)",
-        required=False,
-    )
-
-    corpus_analyse_parser = sub_arg_parsers.add_parser(
-        SUB_CMD_CORPUS_ANALYSE,
-        help="List files in an existing METS corpus that match metadata filters",
-    )
-    corpus_analyse.register_arguments(corpus_analyse_parser)
-
     # evaluate subcommand
     evaluate_arg_parser = sub_arg_parsers.add_parser(
         SUB_CMD_EVALUATE,
@@ -276,24 +196,6 @@ def start() -> None:
         if verbosity > 0:
             print("[INFO ] file_result", file_result)
 
-    elif args.subcommand == SUB_CMD_GROUNDTRUTH_CORPUS:
-        corpus_args = CorpusArgs(
-            input_dir=Path(args.input_dir).absolute(),
-            output_dir=Path(args.output_dir).absolute(),
-            local_cache_dir=Path(args.temp_dir).absolute(),
-            limit=int(args.limit),
-            corpus_label=args.corpus_label,
-            clear_cache=args.clear_cache,
-        )
-        gc.generate(corpus_args)
-
-    elif args.subcommand == SUB_CMD_CORPUS_ANALYSE:
-        analyse_args = vars(args)
-        analyse_args.pop("subcommand", None)
-        result = corpus_analyse.start_analysis(analyse_args)
-        if isinstance(result, corpus_analyse.CorpusCheckResult) and not result.is_valid:
-            raise SystemExit(1)
-
     elif args.subcommand == SUB_CMD_EVALUATE:
         eval_args = vars(args)
         eval_args.pop("subcommand", None)

{ocr_util-2.2.1 → ocr_util-3.0.1}/src/ocr_util/eval/aggregation.py

@@ -142,10 +142,10 @@ def decade_transform(value: str) -> typing.Optional[str]:
 
 
 def century_transform(value: str) -> typing.Optional[str]:
-    """Transform a 4-digit year string into its century bucket (e.g. '1867' → '19th')."""
+    """Bucket years by calendar century: 1801 through 1900 belong to the 19th."""
     try:
         year = int(str(value).strip()[:4])
-        century = year // 100 + 1
+        century = (year - 1) // 100 + 1
         suffix = {1: "st", 2: "nd", 3: "rd"}.get(century % 10 if century % 100 not in (11, 12, 13) else 0, "th")
         return f"{century}{suffix}"
     except (ValueError, TypeError):

{ocr_util-2.2.1 → ocr_util-3.0.1/src/ocr_util.egg-info}/PKG-INFO

@@ -1,11 +1,11 @@
 Metadata-Version: 2.4
 Name: ocr-util
-Version: 2.2.1
+Version: 3.0.1
 Summary: OCR Utils
 Author-email: Universitäts- und Landesbibliothek Sachsen-Anhalt <development@bibliothek.uni-halle.de>
+License-Expression: MIT
 Project-URL: Homepage, https://github.com/ulb-sachsen-anhalt/ocr-util
 Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -38,7 +38,6 @@ Dynamic: license-file
 
 Collection of utils for 
 * evaluation of OCR data for the masses
-* generation of extended OCR-Evaluation Corpora
 * generation of pair-wise Trainingdata for OCR-Backends
 
 ## Requirements
@@ -54,16 +53,10 @@ Each section contains detailed usage help instructions:
 # evaluation
 ocr-util eval --help
 
-# corpus management
-ocr-util corpus --help
-
-# analyse an existing METS corpus
-ocr-util corpus-analyse --help
-
 # slice image by image + input OCR
 ocr-util slice --help
 
-# render image + input OCR
+# render input OCR (regions, lines, words) on given image
 ocr-util show --help
 ```
 
@@ -73,7 +66,7 @@ Inconsistent OCR Groundtruth with empty texts (ALTO String elements missing CONT
 
 _Please note_:  
 Invalid data files are excluded and reported where possible from evaluation.  
-The term 'invalid' refers to errors in schemas in structured XML-data, i.e. syntax errors and further if included geometrical information includes inconsistencies like missing points or missmatching shapes.
+The term 'invalid' refers to malformed or structurally invalid XML, i.e. syntax errors and further if included geometrical information includes inconsistencies like missing points or missmatching shapes.
 
 ### Evaluation Filter-Then-Aggregate
 
@@ -89,6 +82,22 @@ ocr-util eval <candidates> \
 	--aggregate-by "mods:dateIssued:century"
 ```
 
+The dimensions can also be reversed: filter by publication century or decade,
+then aggregate by language. For example, keep only publications from the 17th
+century (1601-1700 inclusive), then group their evaluation results by MODS language:
+
+```bash
+ocr-util eval <candidates> \
+	--reference <groundtruth> \
+	--mets-file <mets.xml> \
+	--filter-by "mods:dateIssued:century=17th" \
+	--aggregate-by "mods:language"
+```
+
+Filtering still happens before evaluation and aggregation; only the metadata
+dimensions exchange roles. To select a decade instead, replace the filter above
+with `--filter-by "mods:dateIssued:decade=1800s"` (1800-1809 inclusive).
+
 Multi-language filter values are interpreted as sets:
 
 ```bash
@@ -100,6 +109,8 @@ ocr-util eval <candidates> \
 ```
 
 Behavior:
+* centuries follow calendar numbering: the 18th century is 1701-1800,
+	and the 19th century is 1801-1900; decades retain ten-year buckets such as 1800-1809
 * single filter value -> exact match (e.g. `ger` does not match `ger+eng`)
 * multi-value filter -> all filter values must be present in any order
 * entries missing the filter criterion are reported as WARNING and discarded
@@ -108,34 +119,6 @@ Behavior:
 * aggregation coverage reports how many evaluated pairs provide each dimension;
 	reported `items` count candidate/ground-truth pairs, not MODS elements
 
-### Corpus Analysis
-
-Existing METS corpora can be filtered without evaluation candidate data. Matching
-full-text file references are printed one per line:
-
-```bash
-ocr-util corpus-analyse <mets.xml> \
-	--filter-by "mods:dateIssued:century=16th"
-```
-
-Repeat `--filter-by` to combine criteria with AND:
-
-```bash
-ocr-util corpus-analyse <mets.xml> \
-	--filter-by "mods:dateIssued:century=16th" \
-	--filter-by "mods:language=ger"
-```
-
-Check that every local file referenced by any METS `FLocat` exists:
-
-```bash
-ocr-util corpus-analyse <mets.xml> --check
-```
-
-Relative paths are resolved from the METS directory. Missing files and remote
-references that cannot be checked locally produce a non-zero exit status. This
-is a file-presence check; it does not perform XML schema validation.
-
 ## Development
 
 Platform: Intel(R) Core(TM) i5-6500 CPU@3.20GHz, 16GB RAM, Ubuntu 22.04 LTS, Python 3.10+

{ocr_util-2.2.1 → ocr_util-3.0.1}/src/ocr_util.egg-info/SOURCES.txt

@@ -9,12 +9,6 @@ src/ocr_util.egg-info/dependency_links.txt
 src/ocr_util.egg-info/entry_points.txt
 src/ocr_util.egg-info/requires.txt
 src/ocr_util.egg-info/top_level.txt
-src/ocr_util/corpus/__init__.py
-src/ocr_util/corpus/analyse.py
-src/ocr_util/corpus/common.py
-src/ocr_util/corpus/generate_corpus.py
-src/ocr_util/corpus/load_metadata.py
-src/ocr_util/corpus/template.corpus.xml
 src/ocr_util/eval/__init__.py
 src/ocr_util/eval/aggregation.py
 src/ocr_util/eval/cli.py
@@ -45,11 +39,6 @@ src/ocr_util/slice/cli.py
 src/ocr_util/slice/gts_pairs.py
 src/ocr_util/slice/pairs_lstmfs.py
 tests/test_aggregation.py
-tests/test_corpus.py
-tests/test_corpus_analyse.py
-tests/test_corpus_examples.py
-tests/test_corpus_fulltext_generation.py
-tests/test_corpus_load_metadata.py
 tests/test_digital_eval_cli.py
 tests/test_digital_object_model.py
 tests/test_digital_object_ocr_alto.py

{ocr_util-2.2.1 → ocr_util-3.0.1}/tests/test_aggregation.py

@@ -697,9 +697,25 @@ def test_century_transform_21st():
     assert digev.century_transform("2026") == "21st"
 
 
-def test_century_transform_boundary_1900():
-    """1900 is in the 20th century."""
-    assert digev.century_transform("1900") == "20th"
+@pytest.mark.parametrize(
+    "year, expected",
+    [
+        ("0001", "1st"),
+        ("0100", "1st"),
+        ("0101", "2nd"),
+        ("1700", "17th"),
+        ("1701", "18th"),
+        ("1800", "18th"),
+        ("1801", "19th"),
+        ("1900", "19th"),
+        ("1901", "20th"),
+        ("2000", "20th"),
+        ("2001", "21st"),
+    ],
+)
+def test_century_transform_calendar_boundaries(year, expected):
+    """Calendar centuries start in year xx01 and end in xx00, not xx00 through xx99."""
+    assert digev.century_transform(year) == expected
 
 
 def test_century_transform_11th_century_no_spurious_st():