topo_map_processor 0.1.9__tar.gz → 0.1.11__tar.gz

{topo_map_processor-0.1.9 → topo_map_processor-0.1.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: topo_map_processor
-Version: 0.1.9
+Version: 0.1.11
 Summary: Library for processing topographic maps
 Project-URL: homepage, https://github.com/ramSeraph/topo_map_processor
 Project-URL: repository, https://github.com/ramSeraph/topo_map_processor.git
@@ -32,7 +32,7 @@ License: This is free and unencumbered software released into the public domain.
 License-File: LICENSE
 Requires-Python: >=3.12
 Requires-Dist: mercantile>=1.2.1
-Requires-Dist: pmtiles>=3.4.1
+Requires-Dist: pmtiles-mosaic>=0.1.0
 Requires-Dist: requests>=2.32.4
 Requires-Dist: shapely>=2.1.1
 Provides-Extra: easyocr
@@ -184,7 +184,13 @@ The `get_intersection_point` method is the most critical part to customize for a
     3.  Identifying the point closest to the expected corner position, while also matching an expected angle and distance from an anchor point.
 
 ![corner for 4way intersection point](/samples/corner3.png)
--   **`get_4way_intersection_point`**: This is a more robust version for crosshair-style corners. It specifically looks for intersections where four lines meet, making it less prone to errors from stray lines in the map collar.
+-   **`get_4way_intersection_point`**: This is a more robust version for crosshair-style corners. It works by:
+    1.  Isolating lines based on their color.
+    2.  Optionally removing any text that might interfere with line detection.
+    3.  Ignoring the edges of the corner image patch to avoid spurious intersections with the border.
+    4.  Detecting all horizontal and vertical lines and finding their intersection points.
+    5.  Validating each intersection by checking for the presence of lines extending in all four directions (up, down, left, right) from the intersection point. This ensures it's a true 4-way "crosshair" intersection.
+    6.  It is designed to be very specific and will fail if it doesn't find exactly one 4-way intersection.
 
 ![corner for biggest contour corner](/samples/corner2.png)
 -   **`get_biggest_contour_corner`**: This strategy is designed for corners that are marked by a solid shape or a filled area (e.g., a colored square or circle). It works by:
@@ -208,15 +214,6 @@ Similar to corner detection, the `locate_grid_lines` method can be implemented u
 
 -   **`get_grid_line_corrections`**: This method serves as a refinement step to improve accuracy. While the transformer provides a very close estimate, this function can correct for minor printing or scanning inaccuracies. For each projected grid intersection, it searches a small area in the actual image for the *true* intersection of printed lines, adjusting the point to match the visual data. This ensures that the lines targeted for removal are precisely the ones printed on the map.
 
-### Tile Sources
-
-The `topo_map_processor.tools.tile_sources` module provides classes for reading tiles from different sources.
-
--   `DiskTilesSource`: Reads tiles from a directory on the local disk.
--   `MBTilesSource`: Reads tiles from an MBTiles file.
--   `PMTilesSource`: Reads tiles from a PMTiles file.
--   `StackedTileSource`: Combines multiple tile sources (Disk, MBTiles, PMTiles) into a single logical source, allowing for seamless access across different storage types.
-
 ### Command-Line Tools
 
 This project provides several command-line tools to work with topographic maps. These tools are dynamically loaded, and you can get the most up-to-date usage information by running the tool with the `--help` flag.
@@ -233,38 +230,6 @@ collect-bounds --bounds-dir <directory> --output-file <file> [--preexisting-file
 -   `--output-file`: Output GeoJSON file path. (Required)
 -   `--preexisting-file`: Pre existing GeoJSON file path. (Optional)
 
-#### `download-mosaic`
-
-Downloads a PMTiles mosaic and converts it to a single MBTiles file. A mosaic is a JSON file that lists multiple PMTiles files.
-
-```bash
-download-mosaic --mosaic_url <url> [--output_file <file>] [--request-timeout-secs <seconds>] [--num-http-retries <retries>]
-```
-
--   `--mosaic_url`, `-u`: URL of the mosaic JSON file. (Required)
--   `--output_file`, `-o`: Output MBTiles file name. If not specified, it is derived from the mosaic URL. (Optional)
--   `--request-timeout-secs`, `-t`: Timeout for HTTP requests in seconds (default: 60).
--   `--num-http-retries`, `-r`: Number of retries for HTTP requests (default: 3).
-
-#### `partition`
-
-Partitions a large set of tiles into smaller PMTiles files. This is useful for managing large datasets, for example, to stay within file size limits for services like GitHub Releases. It creates a mosaic file if the output is partitioned into multiple files.
-
-```bash
-partition --from-source <source> [--from-source <source> ...] --to-pmtiles <file> [--size-limit <limit>]
-```
-
--   `--from-source`: Path to a source file or directory. Can be repeated. Supported sources:
-    -   `.mbtiles` file: e.g., `path/to/file.mbtiles`
-    -   `.pmtiles` file (supports glob patterns): e.g., `path/to/*.pmtiles`
-    -   Directory containing tiles: e.g., `path/to/tiles_dir`
--   `--to-pmtiles`: Output PMTiles file. The prefix for partitioned files will be derived from this argument.
--   `--size-limit`: Maximum size of each partition. Can be a number in bytes (e.g., `100000000`), or with units (e.g., `100M`, `2G`), or a preset:
-    -   `github_release`: 2GB limit
-    -   `github_file`: 100MB limit
-    -   `cloudflare_object`: 512MB limit
-    The overhead (delta) is scaled proportionally based on the `github_release` preset (2GB with 5MB overhead). Defaults to `github_release`.
-
 #### `retile`
 
 Retiles a specific list of sheets (GeoTIFFs) and updates the corresponding tiles in an existing tile set. It can determine which other sheets are affected and need to be pulled in to correctly retile the area.
@@ -285,7 +250,7 @@ retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--tif
 -   `--from-source`: Location of the source from which to pull existing tiles. Can be a glob pattern to match a group of pmtiles. Required if not in pull-list generation mode.
 -   `--sheets-to-pull-list-outfile`: If provided, the script will only calculate the full list of sheets that need to be processed (including adjacent ones) and write it to this file, then exit.
 -   `--num-parallel`: Number of parallel processes to use for tiling (default: number of CPU cores).
--   `--tile-quality`: Quality of compression for webp and jpg (default: 75).
+-   `--tile-quality`: Quality of compression for webp and jpeg (default: 75).
 
 #### `tile`
 
@@ -303,8 +268,8 @@ tile --tiles-dir <dir> --tiffs-dir <dir> --max-zoom <zoom> --name <name> --descr
 -   `--attribution`: Attribution text for the mosaic. Required if `--attribution-file` is not used.
 -   `--attribution-file`: File containing attribution text for the mosaic. Required if `--attribution` is not used.
 -   `--min-zoom`: Minimum zoom level for tiling (default: 0).
--   `--tile-extension`: Tile file extension (default: webp). Choices: `webp`, `jpg`, `jpeg`, `png`.
--   `--tile-quality`: Compression quality of the tiles for `webp` and `jpg` (default: 75).
+-   `--tile-extension`: Tile file extension (default: webp). Choices: `webp`, `jpeg`, `png`.
+-   `--tile-quality`: Compression quality of the tiles for `webp` and `jpeg` (default: 75).
 -   `--num-parallel`: Number of parallel processes to use for tiling (default: number of CPU cores).
 
 ### Running Tools Directly with `uvx`

{topo_map_processor-0.1.9 → topo_map_processor-0.1.11}/README.md

@@ -135,7 +135,13 @@ The `get_intersection_point` method is the most critical part to customize for a
     3.  Identifying the point closest to the expected corner position, while also matching an expected angle and distance from an anchor point.
 
 ![corner for 4way intersection point](/samples/corner3.png)
--   **`get_4way_intersection_point`**: This is a more robust version for crosshair-style corners. It specifically looks for intersections where four lines meet, making it less prone to errors from stray lines in the map collar.
+-   **`get_4way_intersection_point`**: This is a more robust version for crosshair-style corners. It works by:
+    1.  Isolating lines based on their color.
+    2.  Optionally removing any text that might interfere with line detection.
+    3.  Ignoring the edges of the corner image patch to avoid spurious intersections with the border.
+    4.  Detecting all horizontal and vertical lines and finding their intersection points.
+    5.  Validating each intersection by checking for the presence of lines extending in all four directions (up, down, left, right) from the intersection point. This ensures it's a true 4-way "crosshair" intersection.
+    6.  It is designed to be very specific and will fail if it doesn't find exactly one 4-way intersection.
 
 ![corner for biggest contour corner](/samples/corner2.png)
 -   **`get_biggest_contour_corner`**: This strategy is designed for corners that are marked by a solid shape or a filled area (e.g., a colored square or circle). It works by:
@@ -159,15 +165,6 @@ Similar to corner detection, the `locate_grid_lines` method can be implemented u
 
 -   **`get_grid_line_corrections`**: This method serves as a refinement step to improve accuracy. While the transformer provides a very close estimate, this function can correct for minor printing or scanning inaccuracies. For each projected grid intersection, it searches a small area in the actual image for the *true* intersection of printed lines, adjusting the point to match the visual data. This ensures that the lines targeted for removal are precisely the ones printed on the map.
 
-### Tile Sources
-
-The `topo_map_processor.tools.tile_sources` module provides classes for reading tiles from different sources.
-
--   `DiskTilesSource`: Reads tiles from a directory on the local disk.
--   `MBTilesSource`: Reads tiles from an MBTiles file.
--   `PMTilesSource`: Reads tiles from a PMTiles file.
--   `StackedTileSource`: Combines multiple tile sources (Disk, MBTiles, PMTiles) into a single logical source, allowing for seamless access across different storage types.
-
 ### Command-Line Tools
 
 This project provides several command-line tools to work with topographic maps. These tools are dynamically loaded, and you can get the most up-to-date usage information by running the tool with the `--help` flag.
@@ -184,38 +181,6 @@ collect-bounds --bounds-dir <directory> --output-file <file> [--preexisting-file
 -   `--output-file`: Output GeoJSON file path. (Required)
 -   `--preexisting-file`: Pre existing GeoJSON file path. (Optional)
 
-#### `download-mosaic`
-
-Downloads a PMTiles mosaic and converts it to a single MBTiles file. A mosaic is a JSON file that lists multiple PMTiles files.
-
-```bash
-download-mosaic --mosaic_url <url> [--output_file <file>] [--request-timeout-secs <seconds>] [--num-http-retries <retries>]
-```
-
--   `--mosaic_url`, `-u`: URL of the mosaic JSON file. (Required)
--   `--output_file`, `-o`: Output MBTiles file name. If not specified, it is derived from the mosaic URL. (Optional)
--   `--request-timeout-secs`, `-t`: Timeout for HTTP requests in seconds (default: 60).
--   `--num-http-retries`, `-r`: Number of retries for HTTP requests (default: 3).
-
-#### `partition`
-
-Partitions a large set of tiles into smaller PMTiles files. This is useful for managing large datasets, for example, to stay within file size limits for services like GitHub Releases. It creates a mosaic file if the output is partitioned into multiple files.
-
-```bash
-partition --from-source <source> [--from-source <source> ...] --to-pmtiles <file> [--size-limit <limit>]
-```
-
--   `--from-source`: Path to a source file or directory. Can be repeated. Supported sources:
-    -   `.mbtiles` file: e.g., `path/to/file.mbtiles`
-    -   `.pmtiles` file (supports glob patterns): e.g., `path/to/*.pmtiles`
-    -   Directory containing tiles: e.g., `path/to/tiles_dir`
--   `--to-pmtiles`: Output PMTiles file. The prefix for partitioned files will be derived from this argument.
--   `--size-limit`: Maximum size of each partition. Can be a number in bytes (e.g., `100000000`), or with units (e.g., `100M`, `2G`), or a preset:
-    -   `github_release`: 2GB limit
-    -   `github_file`: 100MB limit
-    -   `cloudflare_object`: 512MB limit
-    The overhead (delta) is scaled proportionally based on the `github_release` preset (2GB with 5MB overhead). Defaults to `github_release`.
-
 #### `retile`
 
 Retiles a specific list of sheets (GeoTIFFs) and updates the corresponding tiles in an existing tile set. It can determine which other sheets are affected and need to be pulled in to correctly retile the area.
@@ -236,7 +201,7 @@ retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--tif
 -   `--from-source`: Location of the source from which to pull existing tiles. Can be a glob pattern to match a group of pmtiles. Required if not in pull-list generation mode.
 -   `--sheets-to-pull-list-outfile`: If provided, the script will only calculate the full list of sheets that need to be processed (including adjacent ones) and write it to this file, then exit.
 -   `--num-parallel`: Number of parallel processes to use for tiling (default: number of CPU cores).
--   `--tile-quality`: Quality of compression for webp and jpg (default: 75).
+-   `--tile-quality`: Quality of compression for webp and jpeg (default: 75).
 
 #### `tile`
 
@@ -254,8 +219,8 @@ tile --tiles-dir <dir> --tiffs-dir <dir> --max-zoom <zoom> --name <name> --descr
 -   `--attribution`: Attribution text for the mosaic. Required if `--attribution-file` is not used.
 -   `--attribution-file`: File containing attribution text for the mosaic. Required if `--attribution` is not used.
 -   `--min-zoom`: Minimum zoom level for tiling (default: 0).
--   `--tile-extension`: Tile file extension (default: webp). Choices: `webp`, `jpg`, `jpeg`, `png`.
--   `--tile-quality`: Compression quality of the tiles for `webp` and `jpg` (default: 75).
+-   `--tile-extension`: Tile file extension (default: webp). Choices: `webp`, `jpeg`, `png`.
+-   `--tile-quality`: Compression quality of the tiles for `webp` and `jpeg` (default: 75).
 -   `--num-parallel`: Number of parallel processes to use for tiling (default: number of CPU cores).
 
 ### Running Tools Directly with `uvx`

{topo_map_processor-0.1.9 → topo_map_processor-0.1.11}/pyproject.toml

@@ -9,7 +9,7 @@ authors = [
 ]
 dependencies = [
     "mercantile>=1.2.1",
-    "pmtiles>=3.4.1",
+    "pmtiles-mosaic>=0.1.0",
     "requests>=2.32.4",
     "shapely>=2.1.1",
 ]
@@ -38,13 +38,9 @@ repository = "https://github.com/ramSeraph/topo_map_processor.git"
 
 [project.scripts]
 collect-bounds = "topo_map_processor.tools.collect_bounds:cli"
-download-mosaic = "topo_map_processor.tools.download_mosaic:cli"
-partition = "topo_map_processor.tools.partition:cli"
 retile = "topo_map_processor.tools.retile:cli"
 tile = "topo_map_processor.tools.tile:cli"
-generate-lists = "topo_map_processor.tools.execute_bash:generate_lists"
-upload-to-release = "topo_map_processor.tools.execute_bash:upload_to_release"
-
+retile-e2e = "topo_map_processor.tools.retile_e2e:cli"
 
 [build-system]
 requires = [
@@ -62,12 +58,10 @@ version-file = "topo_map_processor/_version.py"
 [tool.hatch.build.targets.sdist]
 include = [
     "topo_map_processor",
-    "topo_map_processor/bash_scripts/**",
 ]
 
 [tool.hatch.build.targets.wheel]
 include = [
     "topo_map_processor",
-    "topo_map_processor/bash_scripts/**",
 ]
 

{topo_map_processor-0.1.9 → topo_map_processor-0.1.11}/topo_map_processor/_version.py

@@ -17,5 +17,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.1.9'
-__version_tuple__ = version_tuple = (0, 1, 9)
+__version__ = version = '0.1.11'
+__version_tuple__ = version_tuple = (0, 1, 11)

{topo_map_processor-0.1.9 → topo_map_processor-0.1.11}/topo_map_processor/processor.py

@@ -1074,12 +1074,16 @@ class TopoMapProcessor:
 
         return ip
 
-    def get_4way_intersection_point(self, img, line_color, 
+    def get_4way_intersection_point(self, img, line_color, remove_text,
                                     find_line_scale, find_line_iter, 
-                                    cwidth, ext_thresh):
+                                    cwidth, ext_thresh, direction,
+                                    remove_corner_edges_ratio):
         img_mask = self.get_color_mask(img, line_color)
 
-        self.remove_text(img_mask)
+        if remove_text:
+            self.remove_text(img_mask)
+
+        self.remove_corner_edges(img_mask, direction, remove_corner_edges_ratio)
 
         h, w = img_mask.shape[:2]
         self.show_image(img_mask)
@@ -1106,7 +1110,8 @@ class TopoMapProcessor:
             if len(four_corner_ips) > 1:
                 continue
 
-            self.show_points(four_corner_ips, img_mask, [0, 255, 0])
+        self.show_points(four_corner_ips, img_mask, [0, 255, 0])
+
         if len(four_corner_ips) > 1:
             raise Exception(f'multiple intersection points found - {len(four_corner_ips)}')
 

{topo_map_processor-0.1.9 → topo_map_processor-0.1.11}/topo_map_processor/tools/retile.py

@@ -1,16 +1,7 @@
-# /// script
-# requires-python = ">=3.12"
-# dependencies = [
-#     "mercantile",
-#     "pmtiles",
-#     "shapely",
-# ]
-# ///
 
 import os
 import re
 import sys
-import glob
 import json
 import argparse
 
@@ -33,7 +24,7 @@ from osgeo_utils.gdal2tiles import (
     DividedCache,
 )
 
-from .tile_sources import create_source_from_paths, MissingTileError
+from pmtiles_mosaic.tile_sources import create_source_from_paths, MissingTileError
 
 def run_external(cmd):
     print(f'running cmd - {cmd}')
@@ -142,7 +133,7 @@ class Tiler:
                 '--webp-quality', str(self.tile_quality),
             ]
 
-        if self.tile_extension == 'jpg':
+        if self.tile_extension == 'jpeg':
             return params + [
                 '--jpeg_quality', str(self.tile_quality),
             ]
@@ -156,7 +147,7 @@ class Tiler:
         if self.tile_extension == 'webp':
             return 'WEBP'
 
-        if self.tile_extension == 'jpg':
+        if self.tile_extension == 'jpeg':
             return 'JPEG'
 
         if self.tile_extension == 'png':
@@ -171,7 +162,7 @@ class Tiler:
                 'webp_lossless': False,
             }
 
-        if self.tile_extension == 'jpg':
+        if self.tile_extension == 'jpeg':
             return {
                 'jpeg_quality': self.tile_quality,
             }
@@ -308,7 +299,10 @@ class Tiler:
     
             prev_affected_tiles = curr_affected_tiles
     
-        self.disk_handler.save_metadata(self.orig_tile_source.get_metadata())
+        metadata = self.orig_tile_source.get_metadata()
+        metadata['minzoom'] = min_zoom
+        metadata['maxzoom'] = max_zoom
+        self.disk_handler.save_metadata(metadata)
         print('All Done!!!')
 
 
@@ -379,11 +373,7 @@ def assess_sheet_requirements(retile_list_file, bounds_file, max_zoom):
 
     return sheets_to_pull, affected_base_tiles
 
-def cli():
-
-    # I don't care.. this shit isn't worth thinking about
-    if sys.platform == 'darwin':
-        set_start_method('fork')
+def retile_main(args):
 
     parser = argparse.ArgumentParser()
     parser.add_argument('--retile-list-file', required=True, help='File containing list of sheets to retile')
@@ -395,9 +385,9 @@ def cli():
     parser.add_argument('--tiles-dir', help='Directory where the tiles will be created')
     parser.add_argument('--tiffs-dir', help='Directory where the tiffs are present')
     parser.add_argument('--num-parallel', type=int, default=cpu_count(), help='Number of parallel processes to use for tiling (default: number of CPU cores)')
-    parser.add_argument('--tile-quality', default=75, help='quality of compression for webp and jpg (default: 75)')
+    parser.add_argument('--tile-quality', default=75, help='quality of compression for webp and jpeg (default: 75)')
 
-    args = parser.parse_args()
+    args = parser.parse_args(args)
 
     retile_list_file = Path(args.retile_list_file)
     if not retile_list_file.exists():
@@ -452,6 +442,8 @@ def cli():
 
     metadata = orig_source.get_metadata()
     tile_extension = metadata['format']
+    if tile_extension == 'jpg':
+        tile_extension = 'jpeg'
 
     os.environ['GDAL_CACHEMAX'] = '2048'
     os.environ['GDAL_MAX_DATASET_POOL_SIZE'] = '5000'
@@ -470,5 +462,11 @@ def cli():
 
     return 0
 
+def cli():
+    if sys.platform == 'darwin':
+        set_start_method('fork')
+
+    retile_main(sys.argv[1:])
+
 if __name__ == '__main__':
     cli()

topo_map_processor-0.1.11/topo_map_processor/tools/retile_e2e.py

@@ -0,0 +1,182 @@
+
+import argparse
+import csv
+import glob
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from multiprocessing import set_start_method
+
+from pmtiles_mosaic.partition import partition_main
+
+from .retile import retile_main
+
+
+def run_command(cmd, check=True, cwd=None):
+    """Runs a command and prints its output."""
+    print(f"Running command: {' '.join(cmd)}")
+    try:
+        result = subprocess.run(
+            cmd,
+            check=check,
+            capture_output=True,
+            text=True,
+            cwd=cwd,
+        )
+        if result.stdout:
+            print(result.stdout)
+        if result.stderr:
+            print(result.stderr, file=sys.stderr)
+        return result
+    except subprocess.CalledProcessError as e:
+        print(f"Command failed with exit code {e.returncode}", file=sys.stderr)
+        if e.stdout:
+            print("STDOUT:", file=sys.stderr)
+            print(e.stdout, file=sys.stderr)
+        if e.stderr:
+            print("STDERR:", file=sys.stderr)
+            print(e.stderr, file=sys.stderr)
+        raise
+
+
+def cli():
+    """Main function to execute the end-to-end retiling process."""
+
+    if sys.platform == 'darwin':
+        set_start_method('fork')
+
+    parser = argparse.ArgumentParser(description="End-to-end retiling script.")
+    parser.add_argument("-p", "--pmtiles-release", required=True, help="PMTiles release tag")
+    parser.add_argument("-g", "--gtiffs-release", required=True, help="GeoTIFFs release tag")
+    parser.add_argument("-x", "--pmtiles-prefix", required=True, help="Prefix for PMTiles files")
+    parser.add_argument("-l", "--listing-files-tiled", required=True, help="Name of the tiled listing file")
+    args = parser.parse_args()
+
+    # Define paths and filenames
+    sheets_to_pull_list_outfile = Path("sheets_to_pull_list.txt")
+    tiles_dir = Path("staging/tiles/")
+    tiffs_dir = Path("staging/gtiffs/")
+    from_pmtiles_dir = Path("staging/pmtiles")
+    to_pmtiles_dir = Path("export/pmtiles")
+    retile_list_file = Path("to_retile.txt")
+
+    from_pmtiles_prefix = from_pmtiles_dir / args.pmtiles_prefix
+    to_pmtiles_prefix = to_pmtiles_dir / args.pmtiles_prefix
+
+    # Create directories
+    for d in [tiles_dir, tiffs_dir, from_pmtiles_dir, to_pmtiles_dir]:
+        d.mkdir(parents=True, exist_ok=True)
+
+    # Download listing files
+    print("Downloading listing files...")
+    run_command(["gh", "release", "download", args.gtiffs_release, "-p", "listing_files.csv", "--clobber"])
+    run_command(["gh", "release", "download", args.pmtiles_release, "-p", args.listing_files_tiled, "--clobber", "-O", "listing_files_tiled.csv"])
+
+    # Determine sheets to retile
+    print("Determining sheets to retile...")
+    with open("listing_files.csv") as f:
+        gtiffs_sheets = {row['name'] for row in csv.DictReader(f)}
+    with open("listing_files_tiled.csv") as f:
+        pmtiles_sheets = {row['name'] for row in csv.DictReader(f)}
+
+    sheets_to_retile = sorted(list(gtiffs_sheets - pmtiles_sheets))
+    
+    with open(retile_list_file, "w") as f:
+        for sheet in sheets_to_retile:
+            f.write(f"{sheet}\n")
+
+    print(f"Found {len(sheets_to_retile)} sheets to retile.")
+    Path("listing_files_tiled.csv").unlink()
+
+    if not sheets_to_retile:
+        print("No sheets to retile found. Cleaning up and exiting.")
+        Path("listing_files.csv").unlink()
+        retile_list_file.unlink()
+        sys.exit(0)
+
+    # Download bounds and original PMTiles
+    print("Downloading bounds file...")
+    run_command(["gh", "release", "download", args.gtiffs_release, "-p", "bounds.geojson"])
+    print("Getting original PMTiles files...")
+    run_command(["gh", "release", "download", args.pmtiles_release, "-D", str(from_pmtiles_dir), "-p", f"{args.pmtiles_prefix}*"])
+
+    # Get list of sheets to pull
+    print("Getting list of sheets to pull...")
+    retile_main([
+        "--retile-list-file", str(retile_list_file),
+        "--bounds-file", "bounds.geojson",
+        "--sheets-to-pull-list-outfile", str(sheets_to_pull_list_outfile),
+        "--from-source", f"{from_pmtiles_prefix}*.pmtiles",
+    ])
+
+    print("Sheets to pull:")
+    print(sheets_to_pull_list_outfile.read_text())
+
+    # Download TIFFs
+    print("Downloading TIFFs...")
+    with open("listing_files.csv") as f:
+        url_map = {row[0]: row[2] for row in csv.reader(f)}
+
+    with open(sheets_to_pull_list_outfile) as f:
+        for line in f:
+            fname = line.strip()
+            if fname in url_map:
+                url = url_map[fname]
+                print(f"Pulling {fname}")
+                run_command(["wget", "-q", "-P", str(tiffs_dir), url])
+
+    sheets_to_pull_list_outfile.unlink()
+
+    # Retile sheets
+    print("Retiling the sheets...")
+    retile_main([
+        "--retile-list-file", str(retile_list_file),
+        "--bounds-file", "bounds.geojson",
+        "--from-source", f"{from_pmtiles_prefix}*.pmtiles",
+        "--tiles-dir", str(tiles_dir),
+        "--tiffs-dir", str(tiffs_dir),
+    ])
+
+    retile_list_file.unlink()
+    Path("bounds.geojson").unlink()
+
+    # Create new PMTiles files
+    print("Creating new pmtiles files...")
+    partition_main([
+        "--from-source", str(tiles_dir),
+        "--from-source", f"{from_pmtiles_prefix}*.pmtiles",
+        "--to-pmtiles", f"{to_pmtiles_prefix}.pmtiles",
+    ])
+
+    # Upload new PMTiles and manage release assets
+    print("Deleting old PMTiles files from the release...")
+    old_pmtiles = glob.glob(f"{from_pmtiles_dir}/{args.pmtiles_prefix}*")
+    for pmtile in old_pmtiles:
+        run_command(["gh", "release", "delete-asset", args.pmtiles_release, Path(pmtile).name, "-y"])
+
+    print("Uploading new PMTiles files...")
+    new_pmtiles = glob.glob(f"{to_pmtiles_prefix}*")
+    run_command(["gh", "release", "upload", args.pmtiles_release, "--clobber"] + new_pmtiles)
+
+    # Handle listing files
+    if args.listing_files_tiled != "listing_files.csv":
+        print(f"Renaming listing_files.csv to {args.listing_files_tiled}")
+        Path("listing_files.csv").rename(args.listing_files_tiled)
+    
+    print("Uploading new listing file...")
+    run_command(["gh", "release", "upload", args.pmtiles_release, args.listing_files_tiled, "--clobber"])
+    Path(args.listing_files_tiled).unlink()
+
+    # Cleanup
+    print("Cleaning up staging directories...")
+    shutil.rmtree(tiles_dir, ignore_errors=True)
+    shutil.rmtree(tiffs_dir, ignore_errors=True)
+    shutil.rmtree(from_pmtiles_dir, ignore_errors=True)
+    shutil.rmtree(to_pmtiles_dir, ignore_errors=True)
+
+    print("Done!")
+
+
+if __name__ == "__main__":
+    cli()

{topo_map_processor-0.1.9 → topo_map_processor-0.1.11}/topo_map_processor/tools/tile.py

@@ -9,9 +9,12 @@ import subprocess
 
 from pathlib import Path
 from multiprocessing import set_start_method, cpu_count
+from pprint import pprint
 
 from osgeo_utils.gdal2tiles import submain as gdal2tiles_main
 
+SUPPORTED_FORMATS = ['webp', 'jpeg', 'png']
+
 def run_external(cmd):
     print(f'running cmd - {cmd}')
     start = time.time()
@@ -41,7 +44,7 @@ def get_tiler_cmd_params(tile_extension, tile_quality):
             '--webp-quality', str(tile_quality),
         ]
 
-    if tile_extension in ['jpg', 'jpeg']:
+    if tile_extension == 'jpeg':
         return [
             '--tiledriver', 'JPEG',
             '--jpeg_quality', str(tile_quality),
@@ -52,8 +55,16 @@ def get_tiler_cmd_params(tile_extension, tile_quality):
             '--tiledriver', 'PNG',
         ]
 
-    raise ValueError(f'Unsupported tile extension: {tile_extension}')
+    raise ValueError(f'Unsupported tile extension: {tile_extension}, {SUPPORTED_FORMATS=}')
 
+def get_format(tile_extension):
+    if tile_extension == 'webp':
+        return 'webp'
+    if tile_extension == 'jpeg':
+        return 'jpeg'
+    if tile_extension == 'png':
+        return 'png'
+    raise ValueError(f'Unsupported tile extension: {tile_extension}, {SUPPORTED_FORMATS=}')
 
 
 def cli():
@@ -65,7 +76,7 @@ def cli():
     parser.add_argument('--tiffs-dir', required=True, help='Directory with GTiffs to tile')
     parser.add_argument('--max-zoom', type=int, required=True, help='Maximum zoom level for tiling')
     parser.add_argument('--min-zoom', type=int, default=0, help='Minimum zoom level for tiling')
-    parser.add_argument('--tile-extension', type=str, default='webp', choices=['webp', 'jpg', 'jpeg', 'png'], help='Tile file extension (default: webp)')
+    parser.add_argument('--tile-extension', type=str, default='webp', choices=SUPPORTED_FORMATS, help='Tile file extension (default: webp)')
     parser.add_argument('--tile-quality', type=int, default=75, help='Compression quality of the tiles (default: 75)')
     parser.add_argument('--num-parallel', type=int, default=cpu_count(), help='Number of parallel processes to use for tiling (default: number of CPU cores)')
     parser.add_argument('--name', required=True, help='Name of the mosaic.')
@@ -119,7 +130,6 @@ def cli():
         '--resume', 
         '--xyz', 
         '--processes', f'{args.num_parallel}', 
-        '--processes=8', 
         '-z', f'{args.min_zoom}-{args.max_zoom}',
     ] + get_tiler_cmd_params(args.tile_extension, args.tile_quality) + [
         str(vrt_file), str(tiles_dir)
@@ -129,15 +139,13 @@ def cli():
     vrt_file.unlink()
 
     metadata_file = tiles_dir / 'tiles.json'
-    if args.tile_extension == 'jpg':
-        format = 'jpeg'
-    else:
-        format = args.tile_extension
+
+    fmt = get_format(args.tile_extension)
 
     # it is not really a proper tilejson spec.. it is the basic things needed to populate a pmtiles metadata
     metadata = {
         'type': 'baselayer',
-        'format': format,
+        'format': fmt,
         'attribution': attribution_text,
         'description': args.description,
         'name': args.name,
@@ -145,6 +153,9 @@ def cli():
         'maxzoom': args.max_zoom,
         'minzoom': args.min_zoom,
     }
+
+    print(f'writing metadata to {metadata_file}')
+    pprint(metadata)
     metadata_file.write_text(json.dumps(metadata, indent=2))
 
     print('All Done!!')