topo_map_processor 0.1.6__tar.gz → 0.1.7__tar.gz

{topo_map_processor-0.1.6 → topo_map_processor-0.1.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: topo_map_processor
-Version: 0.1.6
+Version: 0.1.7
 Summary: Library for processing topographic maps
 Project-URL: homepage, https://github.com/ramSeraph/topo_map_processor
 Project-URL: repository, https://github.com/ramSeraph/topo_map_processor.git
@@ -31,17 +31,19 @@ License: This is free and unencumbered software released into the public domain.
         For more information, please refer to <https://unlicense.org>
 License-File: LICENSE
 Requires-Python: >=3.12
-Requires-Dist: easyocr>=1.7.2
-Requires-Dist: imgcat>=0.6.0
 Requires-Dist: mercantile>=1.2.1
-Requires-Dist: numpy>=2.3.1
-Requires-Dist: opencv-python-headless>=4.11.0.86
-Requires-Dist: pillow>=11.3.0
 Requires-Dist: pmtiles>=3.4.1
-Requires-Dist: pyproj>=3.7.1
-Requires-Dist: pytesseract>=0.3.13
-Requires-Dist: rasterio>=1.4.3
 Requires-Dist: shapely>=2.1.1
+Provides-Extra: easyocr
+Requires-Dist: easyocr>=1.7.2; extra == 'easyocr'
+Provides-Extra: parse
+Requires-Dist: imgcat>=0.6.0; extra == 'parse'
+Requires-Dist: numpy>=2.3.1; extra == 'parse'
+Requires-Dist: opencv-python-headless>=4.11.0.86; extra == 'parse'
+Requires-Dist: pillow>=11.3.0; extra == 'parse'
+Requires-Dist: pyproj>=3.7.1; extra == 'parse'
+Requires-Dist: pytesseract>=0.3.13; extra == 'parse'
+Requires-Dist: rasterio>=1.4.3; extra == 'parse'
 Description-Content-Type: text/markdown
 
 # Topo Map Processor [![PyPI - Latest Version](https://img.shields.io/pypi/v/topo_map_processor)](https://pypi.org/project/topo_map_processor/) [![GitHub Tag](https://img.shields.io/github/v/tag/ramSeraph/topo_map_processor?filter=v*)](https://github.com/ramSeraph/topo_map_processor/releases/latest)
@@ -119,7 +121,7 @@ The `TopoMapProcessor` class provides a framework for processing individual map
 Here's an example of how to use the `TopoMapProcessor` class.
 
 ```python
-from topo_map_processor import TopoMapProcessor, LineRemovalParams
+from topo_map_processor.processor import TopoMapProcessor, LineRemovalParams
 
 class SampleProcessor(TopoMapProcessor):
 
@@ -205,6 +207,14 @@ Similar to corner detection, the `locate_grid_lines` method can be implemented u
 
 -   **`get_grid_line_corrections`**: This method serves as a refinement step to improve accuracy. While the transformer provides a very close estimate, this function can correct for minor printing or scanning inaccuracies. For each projected grid intersection, it searches a small area in the actual image for the *true* intersection of printed lines, adjusting the point to match the visual data. This ensures that the lines targeted for removal are precisely the ones printed on the map.
 
+### Tile Sources
+
+The `topo_map_processor.tools.tile_sources` module provides classes for reading tiles from different sources.
+
+-   `DiskSource`: Reads tiles from a directory on the local disk.
+-   `PartitionedPMTilesSource`: Reads tiles from a set of partitioned PMTiles files.
+-   `DiskAndPartitionedPMTilesSource`: Reads tiles from both a local directory and a set of partitioned PMTiles files, useful for retiling.
+
 ### Command-Line Tools
 
 This project provides several command-line tools to work with topographic maps. These tools are dynamically loaded, and you can get the most up-to-date usage information by running the tool with the `--help` flag.
@@ -249,7 +259,7 @@ The tool works in two stages. First, it calculates the full set of sheets that n
 In the second stage, it retiles the affected area. It uses the pull list to create a temporary virtual raster (`.vrt`), generates new base tiles for the affected region, and then reconstructs the upper-level tiles by pulling existing, unaffected tiles from the PMTiles source.
 
 ```bash
-retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--min-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-pmtiles-prefix <prefix>] [--sheets-to-pull-list-outfile <file>]
+retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--min-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-pmtiles-prefix <prefix>] [--sheets-to-pull-list-outfile <file>] [--num-parallel <processes>]
 ```
 
 -   `--retile-list-file`: File containing the list of sheets to retile. (Required)
@@ -260,19 +270,21 @@ retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--min
 -   `--tiles-dir`: Directory where the tiles will be created. (Required if not in pull-list generation mode)
 -   `--from-pmtiles-prefix`: Prefix for the PMTiles source from which to pull existing tiles. (Required if not in pull-list generation mode)
 -   `--sheets-to-pull-list-outfile`: If provided, the script will only calculate the full list of sheets that need to be processed (including adjacent ones) and write it to this file, then exit.
+-   `--num-parallel`: Number of parallel processes to use for tiling (default: number of CPU cores).
 
 #### `tile`
 
 Creates web map tiles from a directory of GeoTIFF files. It combines them into a virtual raster (`.vrt`) for efficient processing.
 
 ```bash
-tile <tiles-dir> <tiffs-dir> <max-zoom> [<min-zoom>]
+tile --tiles-dir <dir> --tiffs-dir <dir> --max-zoom <zoom> [--min-zoom <zoom>] [--num-parallel <processes>]
 ```
 
 -   `<tiles-dir>`: Directory to store the output tiles. (Required)
 -   `<tiffs-dir>`: Directory containing the input GeoTIFF files. (Required)
 -   `<max-zoom>`: Maximum zoom level for tiling. (Required)
 -   `<min-zoom>`: Minimum zoom level for tiling (default: 0).
+-   `--num-parallel`: Number of parallel processes to use for tiling (default: number of CPU cores).
 
 ### Running Tools Directly with `uvx`
 

{topo_map_processor-0.1.6 → topo_map_processor-0.1.7}/README.md

@@ -73,7 +73,7 @@ The `TopoMapProcessor` class provides a framework for processing individual map
 Here's an example of how to use the `TopoMapProcessor` class.
 
 ```python
-from topo_map_processor import TopoMapProcessor, LineRemovalParams
+from topo_map_processor.processor import TopoMapProcessor, LineRemovalParams
 
 class SampleProcessor(TopoMapProcessor):
 
@@ -159,6 +159,14 @@ Similar to corner detection, the `locate_grid_lines` method can be implemented u
 
 -   **`get_grid_line_corrections`**: This method serves as a refinement step to improve accuracy. While the transformer provides a very close estimate, this function can correct for minor printing or scanning inaccuracies. For each projected grid intersection, it searches a small area in the actual image for the *true* intersection of printed lines, adjusting the point to match the visual data. This ensures that the lines targeted for removal are precisely the ones printed on the map.
 
+### Tile Sources
+
+The `topo_map_processor.tools.tile_sources` module provides classes for reading tiles from different sources.
+
+-   `DiskSource`: Reads tiles from a directory on the local disk.
+-   `PartitionedPMTilesSource`: Reads tiles from a set of partitioned PMTiles files.
+-   `DiskAndPartitionedPMTilesSource`: Reads tiles from both a local directory and a set of partitioned PMTiles files, useful for retiling.
+
 ### Command-Line Tools
 
 This project provides several command-line tools to work with topographic maps. These tools are dynamically loaded, and you can get the most up-to-date usage information by running the tool with the `--help` flag.
@@ -203,7 +211,7 @@ The tool works in two stages. First, it calculates the full set of sheets that n
 In the second stage, it retiles the affected area. It uses the pull list to create a temporary virtual raster (`.vrt`), generates new base tiles for the affected region, and then reconstructs the upper-level tiles by pulling existing, unaffected tiles from the PMTiles source.
 
 ```bash
-retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--min-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-pmtiles-prefix <prefix>] [--sheets-to-pull-list-outfile <file>]
+retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--min-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-pmtiles-prefix <prefix>] [--sheets-to-pull-list-outfile <file>] [--num-parallel <processes>]
 ```
 
 -   `--retile-list-file`: File containing the list of sheets to retile. (Required)
@@ -214,19 +222,21 @@ retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--min
 -   `--tiles-dir`: Directory where the tiles will be created. (Required if not in pull-list generation mode)
 -   `--from-pmtiles-prefix`: Prefix for the PMTiles source from which to pull existing tiles. (Required if not in pull-list generation mode)
 -   `--sheets-to-pull-list-outfile`: If provided, the script will only calculate the full list of sheets that need to be processed (including adjacent ones) and write it to this file, then exit.
+-   `--num-parallel`: Number of parallel processes to use for tiling (default: number of CPU cores).
 
 #### `tile`
 
 Creates web map tiles from a directory of GeoTIFF files. It combines them into a virtual raster (`.vrt`) for efficient processing.
 
 ```bash
-tile <tiles-dir> <tiffs-dir> <max-zoom> [<min-zoom>]
+tile --tiles-dir <dir> --tiffs-dir <dir> --max-zoom <zoom> [--min-zoom <zoom>] [--num-parallel <processes>]
 ```
 
 -   `<tiles-dir>`: Directory to store the output tiles. (Required)
 -   `<tiffs-dir>`: Directory containing the input GeoTIFF files. (Required)
 -   `<max-zoom>`: Maximum zoom level for tiling. (Required)
 -   `<min-zoom>`: Minimum zoom level for tiling (default: 0).
+-   `--num-parallel`: Number of parallel processes to use for tiling (default: number of CPU cores).
 
 ### Running Tools Directly with `uvx`
 

{topo_map_processor-0.1.6 → topo_map_processor-0.1.7}/pyproject.toml

@@ -8,22 +8,28 @@ authors = [
     { name = "Sreeram Kandimalla", email = "kandimalla.sreeram@gmail.com" },
 ]
 dependencies = [
-    "easyocr>=1.7.2",
-    "imgcat>=0.6.0",
     "mercantile>=1.2.1",
-    "numpy>=2.3.1",
-    "opencv-python-headless>=4.11.0.86",
-    "pillow>=11.3.0",
     "pmtiles>=3.4.1",
-    "pyproj>=3.7.1",
-    "pytesseract>=0.3.13",
-    "rasterio>=1.4.3",
     "shapely>=2.1.1",
 ]
 dynamic = [
     "version"
 ]
 
+[project.optional-dependencies]
+easyocr = [
+    "easyocr>=1.7.2",
+]
+parse = [
+    "numpy>=2.3.1",
+    "pillow>=11.3.0",
+    "imgcat>=0.6.0",
+    "opencv-python-headless>=4.11.0.86",
+    "pyproj>=3.7.1",
+    "pytesseract>=0.3.13",
+    "rasterio>=1.4.3",
+]
+
 [project.urls]
 homepage = "https://github.com/ramSeraph/topo_map_processor"
 repository = "https://github.com/ramSeraph/topo_map_processor.git"
@@ -34,6 +40,8 @@ collect-bounds = "topo_map_processor.tools.collect_bounds:cli"
 partition = "topo_map_processor.tools.partition:cli"
 retile = "topo_map_processor.tools.retile:cli"
 tile = "topo_map_processor.tools.tile:cli"
+generate-lists = "topo_map_processor.tools.execute_bash:generate_lists"
+upload-to-release = "topo_map_processor.tools.execute_bash:upload_to_release"
 
 
 [build-system]
@@ -49,6 +57,15 @@ source = "vcs"
 [tool.hatch.build.hooks.vcs]
 version-file = "topo_map_processor/_version.py"
 
-tool.hatch.build.targets.sdist.include = [
-    "topo_map_processor"
+[tool.hatch.build.targets.sdist]
+include = [
+    "topo_map_processor",
+    "topo_map_processor/bash_scripts/**",
 ]
+
+[tool.hatch.build.targets.wheel]
+include = [
+    "topo_map_processor",
+    "topo_map_processor/bash_scripts/**",
+]
+

{topo_map_processor-0.1.6 → topo_map_processor-0.1.7}/topo_map_processor/_version.py

@@ -17,5 +17,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.1.6'
-__version_tuple__ = version_tuple = (0, 1, 6)
+__version__ = version = '0.1.7'
+__version_tuple__ = version_tuple = (0, 1, 7)

topo_map_processor-0.1.7/topo_map_processor/bash_scripts/generate_lists.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+
+release_base=$1
+ext=$2
+
+if [ -z "$release_base" ] || [ -z "$ext" ]; then
+  echo "Usage: $0 <release_base> <extension>"
+  exit 1
+fi
+
+#owner=$(gh repo view --json owner -q .owner.login)
+#name=$(gh repo view --json name -q .name)
+#repo="$owner/$name"
+
+echo "getting file list"
+echo "name,size,url" > listing_files.csv
+
+# Get all releases, then filter for the base release and any -extra releases.
+releases_to_process=$(gh release list --repo "$repo" --json tagName -q '.[].tagName' | grep -E "^${release_base}(-extra[0-9]+)?$")
+releases_to_process=$(echo "$releases_to_process" | sort -u)
+
+echo "Will process releases: $releases_to_process"
+
+for release in $releases_to_process; do
+  echo "Processing release: $release"
+  gh release view "$release" --json assets -q ".assets[] | select(.name | endswith(\"$ext\")) | \"\\(.name),\\(.size),\\(.url)\"" >> listing_files.csv
+done
+
+gh release upload "$release_base" listing_files.csv --clobber
+
+rm listing_files.csv
+

topo_map_processor-0.1.7/topo_map_processor/bash_scripts/upload_to_release.sh

@@ -0,0 +1,148 @@
+#!/bin/bash
+
+# A script to upload files from a folder to a GitHub release,
+# skipping files that already exist in the release.
+# If the fourth argument is 'yes', it will overwrite existing assets.
+
+set -e
+set -o pipefail
+
+# --- Configuration ---
+# TAG: The git tag of the release to upload to.
+# FOLDER: The local folder containing the files to upload.
+# OVERWRITE: Optional. Set to 'yes' to allow overwriting existing assets.
+
+if [ "$#" -lt 3 ] || [ "$#" -gt 4 ]; then
+    echo "Usage: $0 <tag> <folder> <extension_without_the_leading_dot> [yes_to_overwrite]"
+    exit 1
+fi
+
+TAG="$1"
+FOLDER="$2"
+EXT="$3"
+OVERWRITE_FLAG=${4:-no}
+
+# --- Pre-flight Checks ---
+
+# Check if gh is installed
+if ! command -v gh &> /dev/null; then
+    echo "Error: gh command-line tool is not installed. Please install it to continue."
+    exit 1
+fi
+
+if [ -z "$TAG" ]; then
+    echo "Error: Release tag is not specified."
+    exit 1
+fi
+
+if [ ! -d "$FOLDER" ]; then
+    echo "Error: Folder '$FOLDER' not found."
+    exit 1
+fi
+
+if [ -z "$EXT" ]; then
+    echo "Error: Extension '$EXT' is not specified."
+    exit 1
+fi
+
+ALLOW_OVERWRITE=0
+if [ "$OVERWRITE_FLAG" == "yes" ]; then
+    ALLOW_OVERWRITE=1
+    echo "Overwrite flag is set. Existing assets will be overwritten."
+fi
+
+# --- Main Logic ---
+
+echo "Fetching existing assets for releases matching pattern '${TAG}(-extra[0-9]+)?'..."
+
+# Get all releases, then filter for the base release and any -extra releases, and sort them
+RELEASES_TO_PROCESS=$(gh release list --json tagName -q '.[].tagName' | grep -E "^${TAG}(-extra[0-9]+)?$" | sort -V)
+
+if [ -z "$RELEASES_TO_PROCESS" ]; then
+    echo "Error: No releases found matching pattern '${TAG}(-extra[0-9]+)?'." >&2
+    exit 1
+fi
+
+echo "Found releases: $RELEASES_TO_PROCESS"
+
+# Store assets with their release tags. Format: "asset_name release_tag"
+EXISTING_ASSETS_WITH_RELEASES=""
+for release in $RELEASES_TO_PROCESS; do
+  echo "Fetching assets from release: $release"
+  assets=$(gh release view "$release" --json assets -q '.assets[].name' 2>/dev/null || echo "")
+  if [ -n "$assets" ]; then
+    while IFS= read -r asset_name; do
+      if [ -n "$asset_name" ]; then
+        EXISTING_ASSETS_WITH_RELEASES="${EXISTING_ASSETS_WITH_RELEASES}${asset_name} ${release}"$'
+'
+      fi
+    done <<< "$assets"
+  fi
+done
+
+if [ -z "$EXISTING_ASSETS_WITH_RELEASES" ]; then
+    echo "Could not fetch any assets. This could mean the releases do not have assets, or you lack permissions."
+fi
+
+# Determine available releases and their current asset counts
+AVAILABLE_RELEASES=()
+AVAILABLE_ASSET_COUNTS=()
+for release in $RELEASES_TO_PROCESS; do
+    count=$(gh release view "$release" --json assets -q '.assets | length' 2>/dev/null || echo "0")
+    
+    MAX_ASSETS=998
+    if [ "$release" == "$TAG" ]; then
+        MAX_ASSETS=988
+    fi
+
+    if [ "$count" -lt "$MAX_ASSETS" ]; then
+        AVAILABLE_RELEASES+=("$release")
+        AVAILABLE_ASSET_COUNTS+=("$count")
+    fi
+done
+
+echo "Starting upload process from folder '$FOLDER'..."
+
+# Find all files in the folder that are not yet in any release and loop through them.
+find "${FOLDER}" -type f | grep "^.*\.${EXT}$" | while read -r FILE_PATH; do
+    FILENAME=$(basename "$FILE_PATH")
+
+    RELEASE_FOR_ASSET=$(echo -n "$EXISTING_ASSETS_WITH_RELEASES" | grep "^${FILENAME} " | head -n 1 | awk '{print $2}')
+
+    if [ -n "$RELEASE_FOR_ASSET" ]; then
+        # Asset exists
+        if [ "$ALLOW_OVERWRITE" -eq 1 ]; then
+            echo "  -> Overwriting '$FILENAME' in release '$RELEASE_FOR_ASSET'..."
+            gh release upload "$RELEASE_FOR_ASSET" "$FILE_PATH" --clobber
+        else
+            echo "  -> Skipping '$FILENAME', it already exists in release '$RELEASE_FOR_ASSET'."
+        fi
+        continue
+    fi
+
+    # Find a release to upload to for a new asset
+    if [ ${#AVAILABLE_RELEASES[@]} -eq 0 ]; then
+        echo "Error: All existing releases are full. No space to upload '$FILENAME'." >&2
+        exit 1
+    fi
+
+    UPLOAD_TARGET=${AVAILABLE_RELEASES[0]}
+
+    echo "  -> Uploading '$FILENAME' to '$UPLOAD_TARGET'..."
+    gh release upload "$UPLOAD_TARGET" "$FILE_PATH"
+
+    # Update asset count and remove release from available list if full
+    AVAILABLE_ASSET_COUNTS[0]=$((${AVAILABLE_ASSET_COUNTS[0]} + 1))
+    
+    MAX_ASSETS=998
+    if [ "$UPLOAD_TARGET" == "$TAG" ]; then
+        MAX_ASSETS=988
+    fi
+
+    if [ "${AVAILABLE_ASSET_COUNTS[0]}" -ge "$MAX_ASSETS" ]; then
+        AVAILABLE_RELEASES=("${AVAILABLE_RELEASES[@]:1}")
+        AVAILABLE_ASSET_COUNTS=("${AVAILABLE_ASSET_COUNTS[@]:1}")
+    fi
+done
+
+echo "Upload process complete."

topo_map_processor-0.1.6/topo_map_processor/__init__.py → topo_map_processor-0.1.7/topo_map_processor/processor.py

@@ -14,7 +14,6 @@ import cv2
 import numpy as np
 
 import pytesseract
-import easyocr
 from PIL import Image
 from imgcat import imgcat
 
@@ -551,6 +550,7 @@ class TopoMapProcessor:
         global easy_ocr_reader
     
         if easy_ocr_reader is None:
+            import easyocr
             easy_ocr_reader = easyocr.Reader(['en'], gpu=False)
     
         mask = np.zeros(img.shape[:2], dtype=np.uint8)

topo_map_processor-0.1.7/topo_map_processor/tools/execute_bash.py

@@ -0,0 +1,54 @@
+import subprocess
+import sys
+import shutil
+import importlib.resources as resources
+
+def execute_bash_script(script_name):
+    """
+    Executes a bash script from the bash_scripts directory, locating it
+    relative to the project structure.
+    """
+    if not shutil.which("bash"):
+        raise FileNotFoundError("bash executable not found in PATH")
+
+    # Use importlib.resources to access the script as package data
+    # The script is located in topo_map_processor/bash_scripts
+    try:
+        script_path = resources.files('topo_map_processor.bash_scripts').joinpath(script_name)
+    except FileNotFoundError:
+        raise FileNotFoundError(f"The script {script_name} does not exist or is not a file. Searched in package data.")
+
+    # Execute the bash script and stream output, merging stderr into stdout
+    process = subprocess.Popen(
+        [shutil.which("bash"), str(script_path)] + sys.argv[1:],
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,  # Redirect stderr to stdout
+        text=True,
+        bufsize=1  # Line-buffered
+    )
+
+    # Stream combined stdout and stderr
+    for line in process.stdout:
+        sys.stdout.write(line)
+        sys.stdout.flush()
+
+    process.wait()
+
+    if process.returncode != 0:
+        raise Exception(f"Bash script '{script_name}' failed with exit code {process.returncode}")
+
+def generate_lists():
+    """
+    Executes the generate_lists.sh bash script.
+    """
+    execute_bash_script("generate_lists.sh")
+
+def upload_to_release():
+    """
+    Executes the upload_to_release.sh bash script.
+    """
+    execute_bash_script("upload_to_release.sh")
+
+
+if __name__ == "__main__":
+    generate_lists()

{topo_map_processor-0.1.6 → topo_map_processor-0.1.7}/topo_map_processor/tools/partition.py

@@ -16,7 +16,7 @@ from pprint import pprint
 from pmtiles.tile import zxy_to_tileid, TileType, Compression
 from pmtiles.writer import Writer as PMTilesWriter
 
-from tile_sources import (
+from .tile_sources import (
     DiskSource,
     DiskAndPartitionedPMTilesSource,
     MissingTileError