ocr-stringdist 0.0.6__tar.gz → 0.1.0__tar.gz

ocr_stringdist-0.1.0/CHANGELOG.md

@@ -0,0 +1,16 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2025-04-26
+
+### Added
+
+- Custom insertion and deletion costs for weighted Levenshtein distance.
+
+### Changed
+
+- Breaking changes to Levenshtein distance functions signatures.

{ocr_stringdist-0.0.6 → ocr_stringdist-0.1.0}/Cargo.lock

@@ -74,7 +74,7 @@ dependencies = [
 
 [[package]]
 name = "ocr_stringdist"
-version = "0.0.6"
+version = "0.1.0"
 dependencies = [
  "pyo3",
  "rayon",

{ocr_stringdist-0.0.6 → ocr_stringdist-0.1.0}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "ocr_stringdist"
-version = "0.0.6"
+version = "0.1.0"
 edition = "2021"
 description = "String distances considering OCR errors."
 authors = ["Niklas von Moers <niklasvmoers@protonmail.com>"]

{ocr_stringdist-0.0.6 → ocr_stringdist-0.1.0}/Justfile

@@ -5,10 +5,11 @@ venv:
 
 pytest:
     uv run maturin develop
-    uv run pytest
+    uv run pytest --cov=python/ocr_stringdist tests
 
 test:
-    cargo test
+    cargo llvm-cov
+    #cargo test
 
 mypy:
     uv run mypy .

{ocr_stringdist-0.0.6 → ocr_stringdist-0.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ocr_stringdist
-Version: 0.0.6
+Version: 0.1.0
 Classifier: Programming Language :: Rust
 Classifier: Programming Language :: Python
 Classifier: Operating System :: OS Independent
@@ -40,7 +40,6 @@ pip install ocr-stringdist
 - **Unicode Support**: Arbitrary unicode strings can be compared.
 - **Substitution of Multiple Characters**: Not just character pairs, but string pairs may be substituted, for example the Korean syllable "이" for the two letters "OI".
 - **Pre-defined OCR Distance Map**: A built-in distance map for common OCR confusions (e.g., "0" vs "O", "1" vs "l", "5" vs "S").
-- **Customizable Cost Maps**: Create your own substitution cost maps for specific OCR systems or domains.
 - **Best Match Finder**: Utility function `find_best_candidate` to efficiently find the best matching string from a collection of candidates using any specified distance function (including the library's OCR-aware ones).
 
 ## Usage
@@ -60,7 +59,6 @@ distance = osd.weighted_levenshtein_distance(
     "hi", "Ini",
     cost_map=custom_map,
     symmetric=True,
-    max_token_characters=2,
 )
 print(f"Distance with custom map: {distance}")
 ```

{ocr_stringdist-0.0.6 → ocr_stringdist-0.1.0}/README.md

@@ -25,7 +25,6 @@ pip install ocr-stringdist
 - **Unicode Support**: Arbitrary unicode strings can be compared.
 - **Substitution of Multiple Characters**: Not just character pairs, but string pairs may be substituted, for example the Korean syllable "이" for the two letters "OI".
 - **Pre-defined OCR Distance Map**: A built-in distance map for common OCR confusions (e.g., "0" vs "O", "1" vs "l", "5" vs "S").
-- **Customizable Cost Maps**: Create your own substitution cost maps for specific OCR systems or domains.
 - **Best Match Finder**: Utility function `find_best_candidate` to efficiently find the best matching string from a collection of candidates using any specified distance function (including the library's OCR-aware ones).
 
 ## Usage
@@ -45,7 +44,6 @@ distance = osd.weighted_levenshtein_distance(
     "hi", "Ini",
     cost_map=custom_map,
     symmetric=True,
-    max_token_characters=2,
 )
 print(f"Distance with custom map: {distance}")
 ```

{ocr_stringdist-0.0.6 → ocr_stringdist-0.1.0}/docs/source/api/index.rst

@@ -3,11 +3,10 @@
 API Reference
 =============
 
-This page contains the auto-generated API reference documentation.
-
-.. autofunction:: ocr_stringdist.weighted_levenshtein_distance
-
-.. autofunction:: ocr_stringdist.batch_weighted_levenshtein_distance
+.. automodule:: ocr_stringdist.levenshtein
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 .. automodule:: ocr_stringdist.matching
    :members:

ocr_stringdist-0.1.0/docs/source/changelog.rst

@@ -0,0 +1 @@
+.. mdinclude:: ../../CHANGELOG.md

{ocr_stringdist-0.0.6 → ocr_stringdist-0.1.0}/docs/source/conf.py

@@ -4,6 +4,7 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html
 
 
+import importlib.metadata
 import os
 import sys
 
@@ -17,6 +18,8 @@ sys.path.insert(0, os.path.abspath("../../python"))
 project = "OCR-StringDist"
 copyright = "2025, Niklas von Moers"
 author = "Niklas von Moers"
+release = importlib.metadata.version("ocr_stringdist")
+version = release
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

ocr_stringdist-0.1.0/docs/source/examples.rst

@@ -0,0 +1,20 @@
+==========
+ Examples
+==========
+
+.. code-block:: python
+
+    import ocr_stringdist as osd
+
+    # Using default OCR distance map
+    distance = osd.weighted_levenshtein_distance("OCR5", "OCRS")
+    print(f"Distance between 'OCR5' and 'OCRS': {distance}")  # Will be less than 1.0
+
+    # Custom cost map
+    custom_map = {("In", "h"): 0.5}
+    distance = osd.weighted_levenshtein_distance(
+        "hi", "Ini",
+        cost_map=custom_map,
+        symmetric=True,
+    )
+    print(f"Distance with custom map: {distance}")

ocr_stringdist-0.1.0/docs/source/getting-started.rst

@@ -0,0 +1,10 @@
+=================
+ Getting Started
+=================
+
+Installation
+============
+
+.. code-block:: console
+
+    pip install ocr-stringdist

ocr_stringdist-0.1.0/docs/source/index.rst

@@ -0,0 +1,36 @@
+================
+ OCR-StringDist
+================
+
+A Python library for string distance calculations that account for common OCR (optical character recognition) errors, written in Rust.
+
+:Repository: https://niklasvonm.github.io/ocr-stringdist/
+:Current version: |release|
+
+.. image:: https://img.shields.io/badge/PyPI-Package-blue
+   :target: https://pypi.org/project/ocr-stringdist/
+   :alt: PyPI
+
+.. image:: https://img.shields.io/badge/License-MIT-green
+   :target: LICENSE
+   :alt: License
+
+Features
+========
+
+- **Weighted Levenshtein Distance**: An adaptation of the classic Levenshtein algorithm with custom substitution costs for character pairs that are commonly confused in OCR models, including efficient batch processing.
+- **Unicode Support**: Arbitrary unicode strings can be compared.
+- **Substitution of Multiple Characters**: Not just character pairs, but string pairs may be substituted, for example the Korean syllable "이" for the two letters "OI".
+- **Pre-defined OCR Distance Map**: A built-in distance map for common OCR confusions (e.g., "0" vs "O", "1" vs "l", "5" vs "S").
+- **Best Match Finder**: Utility function ``find_best_candidate`` to efficiently find the best matching string from a collection of candidates using any specified distance function (including the library's OCR-aware ones).
+
+Contents
+========
+
+.. toctree::
+   :maxdepth: 1
+
+   getting-started
+   examples
+   api/index
+   changelog

{ocr_stringdist-0.0.6 → ocr_stringdist-0.1.0}/examples/batch_processing.py

@@ -8,8 +8,6 @@ from typing import Any, Callable
 
 import ocr_stringdist as osd
 
-MAX_TOKEN_CHARACTERS = 1
-
 
 def benchmark(func: Callable, *args: Any, **kwargs: Any) -> tuple[Any, float]:  # type: ignore
     """Run a function and return the execution time in seconds."""
@@ -32,12 +30,7 @@ def compare_methods() -> None:
 
     # Standard loop approach
     _, time_loop = benchmark(
-        lambda: [
-            osd.weighted_levenshtein_distance(
-                source, cand, max_token_characters=MAX_TOKEN_CHARACTERS
-            )
-            for cand in candidates
-        ]
+        lambda: [osd.weighted_levenshtein_distance(source, cand) for cand in candidates]
     )
     print(
         f"Loop of single calls: {time_loop:.6f} seconds "
@@ -45,12 +38,7 @@ def compare_methods() -> None:
     )
 
     # Batch approach
-    _, time_batch = benchmark(
-        osd.batch_weighted_levenshtein_distance,
-        source,
-        candidates,
-        max_token_characters=MAX_TOKEN_CHARACTERS,
-    )
+    _, time_batch = benchmark(osd.batch_weighted_levenshtein_distance, source, candidates)
     print(
         f"Batch function: {time_batch:.6f} seconds "
         f"({1000 * time_batch / len(candidates):.6f}ms each)"

{ocr_stringdist-0.0.6 → ocr_stringdist-0.1.0}/examples/weighted_levenshtein.py

@@ -24,7 +24,6 @@ ic(
         "이탈리",
         "OI탈리",  # Korean syllables may be confused with multiple Latin letters at once
         {("이", "OI"): 0.5},
-        max_token_characters=2,
     ),
 )
 
@@ -32,13 +31,13 @@ ic(
     weighted_levenshtein_distance(
         "ABCDE",
         "XBCDE",
-        cost_map={},
-        default_cost=0.8,  # Lower default substitution cost (default is 1.0)
+        substitution_costs={},
+        default_substitution_cost=0.8,  # Lower default substitution cost (default is 1.0)
     )
 )
 
-ic(weighted_levenshtein_distance("A", "B", {("A", "B"): 0.0}, symmetric=False))
-ic(weighted_levenshtein_distance("A", "B", {("B", "A"): 0.0}, symmetric=False))
+ic(weighted_levenshtein_distance("A", "B", {("A", "B"): 0.0}, symmetric_substitution=False))
+ic(weighted_levenshtein_distance("A", "B", {("B", "A"): 0.0}, symmetric_substitution=False))
 
 ic(
     find_best_candidate(

{ocr_stringdist-0.0.6 → ocr_stringdist-0.1.0}/pyproject.toml

@@ -27,6 +27,7 @@ dev = [
     "maturin>=1.8.3",
     "mypy>=1.15.0",
     "pytest>=8.3.5",
+    "pytest-cov>=6.1.1",
     "ruff>=0.11.6",
     "wheel>=0.45.1",
 ]

ocr_stringdist-0.1.0/python/ocr_stringdist/__init__.py

@@ -0,0 +1,10 @@
+from .default_ocr_distances import ocr_distance_map
+from .levenshtein import batch_weighted_levenshtein_distance, weighted_levenshtein_distance
+from .matching import find_best_candidate
+
+__all__ = [
+    "ocr_distance_map",
+    "weighted_levenshtein_distance",
+    "batch_weighted_levenshtein_distance",
+    "find_best_candidate",
+]

ocr_stringdist-0.1.0/python/ocr_stringdist/levenshtein.py

@@ -0,0 +1,118 @@
+from typing import Optional
+
+from ._rust_stringdist import *  # noqa: F403
+from .default_ocr_distances import ocr_distance_map
+
+
+def weighted_levenshtein_distance(
+    s1: str,
+    s2: str,
+    /,
+    substitution_costs: Optional[dict[tuple[str, str], float]] = None,
+    insertion_costs: Optional[dict[str, float]] = None,
+    deletion_costs: Optional[dict[str, float]] = None,
+    *,
+    symmetric_substitution: bool = True,
+    default_substitution_cost: float = 1.0,
+    default_insertion_cost: float = 1.0,
+    default_deletion_cost: float = 1.0,
+) -> float:
+    """
+    Levenshtein distance with custom substitution, insertion and deletion costs.
+
+    The default `substitution_costs` considers common OCR errors, see
+    :py:data:`ocr_stringdist.default_ocr_distances.ocr_distance_map`.
+
+    :param s1: First string (interpreted as the string read via OCR)
+    :param s2: Second string
+    :param substitution_costs: Dictionary mapping tuples of strings ("substitution tokens") to their
+                     substitution costs. Only one direction needs to be configured unless
+                     `symmetric_substitution` is False.
+                     Note that the runtime scales in the length of the longest substitution token.
+                     Defaults to `ocr_stringdist.ocr_distance_map`.
+    :param insertion_costs: Dictionary mapping strings to their insertion costs.
+    :param deletion_costs: Dictionary mapping strings to their deletion costs.
+    :param symmetric_substitution: Should the keys of `substitution_costs` be considered to be
+                                   symmetric? Defaults to True.
+    :param default_substitution_cost: The default substitution cost for character pairs not found
+                                      in `substitution_costs`.
+    :param default_insertion_cost: The default insertion cost for characters not found in
+                                   `insertion_costs`.
+    :param default_deletion_cost: The default deletion cost for characters not found in
+                                  `deletion_costs`.
+    """
+    if substitution_costs is None:
+        substitution_costs = ocr_distance_map
+    if insertion_costs is None:
+        insertion_costs = {}
+    if deletion_costs is None:
+        deletion_costs = {}
+    # _weighted_levenshtein_distance is written in Rust, see src/rust_stringdist.rs.
+    return _weighted_levenshtein_distance(  # type: ignore  # noqa: F405
+        s1,
+        s2,
+        substitution_costs=substitution_costs,
+        insertion_costs=insertion_costs,
+        deletion_costs=deletion_costs,
+        symmetric_substitution=symmetric_substitution,
+        default_substitution_cost=default_substitution_cost,
+        default_insertion_cost=default_insertion_cost,
+        default_deletion_cost=default_deletion_cost,
+    )
+
+
+def batch_weighted_levenshtein_distance(
+    s: str,
+    candidates: list[str],
+    /,
+    substitution_costs: Optional[dict[tuple[str, str], float]] = None,
+    insertion_costs: Optional[dict[str, float]] = None,
+    deletion_costs: Optional[dict[str, float]] = None,
+    *,
+    symmetric_substitution: bool = True,
+    default_substitution_cost: float = 1.0,
+    default_insertion_cost: float = 1.0,
+    default_deletion_cost: float = 1.0,
+) -> list[float]:
+    """
+    Calculate weighted Levenshtein distances between a string and multiple candidates.
+
+    This is more efficient than calling :func:`weighted_levenshtein_distance` multiple times.
+
+    :param s: The string to compare (interpreted as the string read via OCR)
+    :param candidates: List of candidate strings to compare against
+    :param substitution_costs: Dictionary mapping tuples of strings ("substitution tokens") to their
+                     substitution costs. Only one direction needs to be configured unless
+                     `symmetric_substitution` is False.
+                     Note that the runtime scales in the length of the longest substitution token.
+                     Defaults to `ocr_stringdist.ocr_distance_map`.
+    :param insertion_costs: Dictionary mapping strings to their insertion costs.
+    :param deletion_costs: Dictionary mapping strings to their deletion costs.
+    :param symmetric_substitution: Should the keys of `substitution_costs` be considered to be
+                                   symmetric? Defaults to True.
+    :param default_substitution_cost: The default substitution cost for character pairs not found
+                                      in `substitution_costs`.
+    :param default_insertion_cost: The default insertion cost for characters not found in
+                                   `insertion_costs`.
+    :param default_deletion_cost: The default deletion cost for characters not found in
+                                  `deletion_costs`.
+    :return: A list of distances corresponding to each candidate
+    """
+    if substitution_costs is None:
+        substitution_costs = ocr_distance_map
+    if insertion_costs is None:
+        insertion_costs = {}
+    if deletion_costs is None:
+        deletion_costs = {}
+    # _batch_weighted_levenshtein_distance is written in Rust, see src/rust_stringdist.rs.
+    return _batch_weighted_levenshtein_distance(  # type: ignore  # noqa: F405
+        s,
+        candidates,
+        substitution_costs=substitution_costs,
+        insertion_costs=insertion_costs,
+        deletion_costs=deletion_costs,
+        symmetric_substitution=symmetric_substitution,
+        default_substitution_cost=default_substitution_cost,
+        default_insertion_cost=default_insertion_cost,
+        default_deletion_cost=default_deletion_cost,
+    )