topo_map_processor 0.1.13__tar.gz → 0.1.16__tar.gz

{topo_map_processor-0.1.13 → topo_map_processor-0.1.16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: topo_map_processor
-Version: 0.1.13
+Version: 0.1.16
 Summary: Library for processing topographic maps
 Project-URL: homepage, https://github.com/ramSeraph/topo_map_processor
 Project-URL: repository, https://github.com/ramSeraph/topo_map_processor.git
@@ -45,6 +45,32 @@ Requires-Dist: pillow>=11.3.0; extra == 'parse'
 Requires-Dist: pyproj>=3.7.1; extra == 'parse'
 Requires-Dist: pytesseract>=0.3.13; extra == 'parse'
 Requires-Dist: rasterio>=1.4.3; extra == 'parse'
+Provides-Extra: pinned
+Requires-Dist: aiofiles==24.1.0; extra == 'pinned'
+Requires-Dist: aiohappyeyeballs==2.6.1; extra == 'pinned'
+Requires-Dist: aiohttp==3.12.15; extra == 'pinned'
+Requires-Dist: aiosignal==1.4.0; extra == 'pinned'
+Requires-Dist: attrs==25.3.0; extra == 'pinned'
+Requires-Dist: certifi==2025.8.3; extra == 'pinned'
+Requires-Dist: charset-normalizer==3.4.3; extra == 'pinned'
+Requires-Dist: click==8.3.0; extra == 'pinned'
+Requires-Dist: colorama==0.4.6; (sys_platform == 'win32') and extra == 'pinned'
+Requires-Dist: colorlog==6.9.0; extra == 'pinned'
+Requires-Dist: frozenlist==1.7.0; extra == 'pinned'
+Requires-Dist: idna==3.10; extra == 'pinned'
+Requires-Dist: mercantile==1.2.1; extra == 'pinned'
+Requires-Dist: multidict==6.6.4; extra == 'pinned'
+Requires-Dist: numpy==2.3.3; extra == 'pinned'
+Requires-Dist: pillow==11.3.0; extra == 'pinned'
+Requires-Dist: pmtiles-mosaic==0.1.5; extra == 'pinned'
+Requires-Dist: pmtiles==3.4.1; extra == 'pinned'
+Requires-Dist: propcache==0.3.2; extra == 'pinned'
+Requires-Dist: pypdl==1.5.6; extra == 'pinned'
+Requires-Dist: requests==2.32.5; extra == 'pinned'
+Requires-Dist: shapely==2.1.1; extra == 'pinned'
+Requires-Dist: typing-extensions==4.15.0; (python_full_version < '3.13') and extra == 'pinned'
+Requires-Dist: urllib3==2.5.0; extra == 'pinned'
+Requires-Dist: yarl==1.20.1; extra == 'pinned'
 Description-Content-Type: text/markdown
 
 # Topo Map Processor [![PyPI - Latest Version](https://img.shields.io/pypi/v/topo_map_processor)](https://pypi.org/project/topo_map_processor/) [![GitHub Tag](https://img.shields.io/github/v/tag/ramSeraph/topo_map_processor?filter=v*)](https://github.com/ramSeraph/topo_map_processor/releases/latest)
@@ -239,11 +265,12 @@ The tool works in two stages. First, it calculates the full set of sheets that n
 In the second stage, it retiles the affected area. It uses the pull list to create a temporary virtual raster (`.vrt`), generates new base tiles for the affected region, and then reconstructs the upper-level tiles by pulling existing, unaffected tiles from the PMTiles source.
 
 ```bash
-retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-source <source>] [--sheets-to-pull-list-outfile <file>] [--num-parallel <processes>] [--tile-quality <quality>]
+retile --retile-list-file <file> --bounds-file <file> [--dummy-bounds-file <file>] [--max-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-source <source>] [--sheets-to-pull-list-outfile <file>] [--num-parallel <processes>] [--tile-quality <quality>]
 ```
 
 -   `--retile-list-file`: File containing the list of sheets to retile. (Required)
 -   `--bounds-file`: GeoJSON file containing the list of available sheets and their geographic bounds. (Required)
+-   `--dummy-bounds-file`: GeoJSON file containing geographic bounds of areas that need to be updated. These areas might not have any new sheets. (Optional)
 -   `--max-zoom`: Maximum zoom level to create tiles for. Defaults to the max zoom from the source if `--from-source` is provided, otherwise it is required.
 -   `--tiffs-dir`: Directory where the GeoTIFFs are located. (Required if not in pull-list generation mode)
 -   `--tiles-dir`: Directory where the tiles will be created. (Required if not in pull-list generation mode)

{topo_map_processor-0.1.13 → topo_map_processor-0.1.16}/README.md

@@ -190,11 +190,12 @@ The tool works in two stages. First, it calculates the full set of sheets that n
 In the second stage, it retiles the affected area. It uses the pull list to create a temporary virtual raster (`.vrt`), generates new base tiles for the affected region, and then reconstructs the upper-level tiles by pulling existing, unaffected tiles from the PMTiles source.
 
 ```bash
-retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-source <source>] [--sheets-to-pull-list-outfile <file>] [--num-parallel <processes>] [--tile-quality <quality>]
+retile --retile-list-file <file> --bounds-file <file> [--dummy-bounds-file <file>] [--max-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-source <source>] [--sheets-to-pull-list-outfile <file>] [--num-parallel <processes>] [--tile-quality <quality>]
 ```
 
 -   `--retile-list-file`: File containing the list of sheets to retile. (Required)
 -   `--bounds-file`: GeoJSON file containing the list of available sheets and their geographic bounds. (Required)
+-   `--dummy-bounds-file`: GeoJSON file containing geographic bounds of areas that need to be updated. These areas might not have any new sheets. (Optional)
 -   `--max-zoom`: Maximum zoom level to create tiles for. Defaults to the max zoom from the source if `--from-source` is provided, otherwise it is required.
 -   `--tiffs-dir`: Directory where the GeoTIFFs are located. (Required if not in pull-list generation mode)
 -   `--tiles-dir`: Directory where the tiles will be created. (Required if not in pull-list generation mode)

{topo_map_processor-0.1.13 → topo_map_processor-0.1.16}/pyproject.toml

@@ -30,6 +30,33 @@ parse = [
     "pytesseract>=0.3.13",
     "rasterio>=1.4.3",
 ]
+pinned = [
+    "aiofiles==24.1.0",
+    "aiohappyeyeballs==2.6.1",
+    "aiohttp==3.12.15",
+    "aiosignal==1.4.0",
+    "attrs==25.3.0",
+    "certifi==2025.8.3",
+    "charset-normalizer==3.4.3",
+    "click==8.3.0",
+    "colorama==0.4.6 ; sys_platform == 'win32'",
+    "colorlog==6.9.0",
+    "frozenlist==1.7.0",
+    "idna==3.10",
+    "mercantile==1.2.1",
+    "multidict==6.6.4",
+    "numpy==2.3.3",
+    "pillow==11.3.0",
+    "pmtiles==3.4.1",
+    "pmtiles-mosaic==0.1.5",
+    "propcache==0.3.2",
+    "pypdl==1.5.6",
+    "requests==2.32.5",
+    "shapely==2.1.1",
+    "typing-extensions==4.15.0 ; python_full_version < '3.13'",
+    "urllib3==2.5.0",
+    "yarl==1.20.1",
+]
 
 [project.urls]
 homepage = "https://github.com/ramSeraph/topo_map_processor"

{topo_map_processor-0.1.13 → topo_map_processor-0.1.16}/topo_map_processor/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.1.13'
-__version_tuple__ = version_tuple = (0, 1, 13)
+__version__ = version = '0.1.16'
+__version_tuple__ = version_tuple = (0, 1, 16)
 
 __commit_id__ = commit_id = None

{topo_map_processor-0.1.13 → topo_map_processor-0.1.16}/topo_map_processor/tools/retile.py

@@ -4,6 +4,7 @@ import re
 import sys
 import json
 import argparse
+import tempfile
 
 import time
 import subprocess
@@ -185,7 +186,8 @@ class Tiler:
             '-r', 'antialias',
             '--verbose',
             '-w', 'none',
-            '--exclude', 
+            # N.B. if transperent tiles are excluded, the old data might show through
+            #'--exclude',
             '--resume', 
             '--xyz', 
             '--processes', f'{self.pool._processes}', 
@@ -205,7 +207,8 @@ class Tiler:
             'verbose': False,
             'quiet': False,
             'xyz': True,
-            'exclude_transparent': True,
+            # N.B. if transperent tiles are excluded, the old data might show through
+            #'exclude_transparent': True,
             'profile': 'mercator',
             'resampling': 'antialias',
             'tiledriver': tile_driver,
@@ -319,7 +322,7 @@ def check_sheets(sheets_to_pull, tiffs_dir):
         raise Exception(f'missing files {missing}')
 
 
-def get_sheet_data(bounds_fname):
+def get_sheet_data(bounds_fname, dummy=False):
 
     index_file = Path(bounds_fname)
     if not index_file.exists():
@@ -327,9 +330,14 @@ def get_sheet_data(bounds_fname):
 
     index_data = json.loads(index_file.read_text())
 
+    dummy_count = 0
     sheets_to_box = {}
     for f in index_data['features']:
-        sheet_no = f['properties']['id']
+        if not dummy:
+            sheet_no = f['properties']['id']
+        else:
+            sheet_no = f'dummy_{dummy_count}'
+            dummy_count += 1
         geom = shape(f['geometry'])
         xmin, ymin, xmax, ymax = geom.bounds
         box = mercantile.LngLatBbox(xmin, ymin, xmax, ymax)
@@ -354,33 +362,61 @@ def get_base_tile_sheet_mappings(sheets_to_box, base_zoom):
 
    
 
-def assess_sheet_requirements(retile_list_file, bounds_file, max_zoom):
+def assess_sheet_requirements(retile_list_file, bounds_file, force_redo_bounds_file, max_zoom):
 
     retile_sheets = retile_list_file.read_text().split('\n')
     retile_sheets = set([ r.strip().replace('.tif', '') for r in retile_sheets if r.strip() != '' ])
 
     print('getting base tiles to sheet mapping')
     sheets_to_box = get_sheet_data(bounds_file)
+    force_redo_sheets_to_box = {}
+    if force_redo_bounds_file is not None:
+        force_redo_sheets_to_box = get_sheet_data(force_redo_bounds_file, dummy=True)
+        sheets_to_box.update(force_redo_sheets_to_box)
+
     sheets_to_base_tiles, base_tiles_to_sheets = get_base_tile_sheet_mappings(sheets_to_box, max_zoom)
 
     print('calculating sheets to pull')
     affected_base_tiles = set()
     for sheet_no in retile_sheets:
         affected_base_tiles.update(sheets_to_base_tiles[sheet_no])
+    for sheet_no in force_redo_sheets_to_box.keys():
+        affected_base_tiles.update(sheets_to_base_tiles[sheet_no])
 
     sheets_to_pull = set()
     for tile in affected_base_tiles:
         to_add = base_tiles_to_sheets[tile]
         for sheet in to_add:
-            sheets_to_pull.add(sheet + '.tif')
+            if sheet not in force_redo_sheets_to_box:
+                sheets_to_pull.add(sheet + '.tif')
 
     return sheets_to_pull, affected_base_tiles
 
+def create_dummy_sheets(dummy_bounds_file, tiffs_dir):
+    print('creating dummy sheets for force redo areas')
+    dummy_sheets_to_box = get_sheet_data(dummy_bounds_file, dummy=True)
+
+    for sheet_no, box in dummy_sheets_to_box.items():
+        file = tiffs_dir.joinpath(f'{sheet_no}.tif')
+        if file.exists():
+            continue
+
+        west, south, east, north = box.west, box.south, box.east, box.north
+        temp_file = Path(tempfile.mktemp(suffix='.tif'))
+        cmd = f'gdal_create -of GTiff -a_srs EPSG:4326 -a_ullr {west} {north} {east} {south} -a_nodata 0 -ot Byte -co COMPRESS=LZW -bands 3 -burn 0 0 0 -outsize 256 256 {temp_file}'
+        run_external(cmd)
+        temp_file_1 = Path(tempfile.mktemp(suffix='.tif'))
+        cmd = f'gdalwarp -dstalpha -t_srs EPSG:3857 -co COMPRESS=LZW {temp_file} {temp_file_1}'
+        run_external(cmd)
+        cmd = f'gdal_translate -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=JPEG -co QUALITY=100 --config GDAL_TIFF_INTERNAL_MASK YES  -b 1 -b 2 -b 3 -mask 4 --config GDAL_CACHEMAX 512 -of COG {temp_file_1} {file}'
+        run_external(cmd)
+
 def retile_main(args):
 
     parser = argparse.ArgumentParser()
     parser.add_argument('--retile-list-file', required=True, help='File containing list of sheets to retile')
     parser.add_argument('--bounds-file', required=True, help='Geojson file containing list of available sheets and their georaphic bounds')
+    parser.add_argument('--force-redo-bounds-file', required=False, help='Geojson file containing georaphic bounds of areas that need to be updated, these areas might not have any new sheets')
     parser.add_argument('--max-zoom', type=int, help='Maximum zoom level to create tiles for, needed only when --from-source is not provided')
     parser.add_argument('--sheets-to-pull-list-outfile', default=None,
                         help='Output into which we write the list of sheet that need to be pulled, if set, the script ends after it created the list file')
@@ -400,6 +436,12 @@ def retile_main(args):
     if not bounds_file.exists():
         parser.error(f'Bounds file {args.bounds_file} does not exist')
 
+    force_redo_bounds_file = None
+    if args.force_redo_bounds_file:
+        force_redo_bounds_file = Path(args.force_redo_bounds_file) 
+        if not force_redo_bounds_file.exists():
+            parser.error(f'Force redo bounds file {args.force_redo_bounds_file} does not exist')
+
 
     if not args.sheets_to_pull_list_outfile:
         missing = []
@@ -430,7 +472,7 @@ def retile_main(args):
             print(f'--max-zoom argument is ignored when --from-source is set, using max zoom from original source: {max_zoom}')
 
     # calculate the sheets to pull and affected base tiles
-    sheets_to_pull, affected_base_tiles = assess_sheet_requirements(retile_list_file, bounds_file, max_zoom)
+    sheets_to_pull, affected_base_tiles = assess_sheet_requirements(retile_list_file, bounds_file, force_redo_bounds_file, max_zoom)
 
     if args.sheets_to_pull_list_outfile is not None:
         Path(args.sheets_to_pull_list_outfile).write_text('\n'.join(sheets_to_pull) + '\n')
@@ -443,6 +485,9 @@ def retile_main(args):
     print('check the sheets availability')
     check_sheets(sheets_to_pull, tiffs_dir)
 
+    if force_redo_bounds_file is not None:
+        create_dummy_sheets(force_redo_bounds_file, tiffs_dir)
+
     metadata = orig_source.get_metadata()
     tile_extension = metadata['format']
     if tile_extension == 'jpg':

{topo_map_processor-0.1.13 → topo_map_processor-0.1.16}/topo_map_processor/tools/retile_e2e.py

@@ -73,6 +73,15 @@ def cli():
     run_command(["gh", "release", "download", args.gtiffs_release, "-p", "listing_files.csv", "--clobber"])
     run_command(["gh", "release", "download", args.pmtiles_release, "-p", args.listing_files_tiled, "--clobber", "-O", "listing_files_tiled.csv"])
 
+    print("Download force redo bounds")
+    force_redo_bounds_file = Path("force_redo_bounds.geojson")
+    try:
+        run_command(["gh", "release", "download", args.pmtiles_release, "-p", f"{force_redo_bounds_file}", "--clobber"])
+        print("Force redo bounds file found and downloaded.")
+    except subprocess.CalledProcessError:
+        print("No force redo bounds file found; proceeding without it.")
+        force_redo_bounds_file = None
+
     # Determine sheets to retile
     print("Determining sheets to retile...")
     with open("listing_files.csv") as f:
@@ -101,6 +110,9 @@ def cli():
     print("Getting original PMTiles files...")
     run_command(["gh", "release", "download", args.pmtiles_release, "-D", str(from_pmtiles_dir), "-p", f"{args.pmtiles_prefix}*"])
 
+    extra_bounds_args = []
+    if force_redo_bounds_file:
+        extra_bounds_args = ["--force-redo-bounds-file", str(force_redo_bounds_file)]
     # Get list of sheets to pull
     print("Getting list of sheets to pull...")
     retile_main([
@@ -108,7 +120,7 @@ def cli():
         "--bounds-file", "bounds.geojson",
         "--sheets-to-pull-list-outfile", str(sheets_to_pull_list_outfile),
         "--from-source", f"{from_pmtiles_prefix}*.pmtiles",
-    ])
+    ] + extra_bounds_args)
 
     print("Sheets to pull:")
     print(sheets_to_pull_list_outfile.read_text())
@@ -136,7 +148,7 @@ def cli():
         "--from-source", f"{from_pmtiles_prefix}*.pmtiles",
         "--tiles-dir", str(tiles_dir),
         "--tiffs-dir", str(tiffs_dir),
-    ])
+    ] + extra_bounds_args)
 
     retile_list_file.unlink()
     Path("bounds.geojson").unlink()
@@ -168,6 +180,12 @@ def cli():
     run_command(["gh", "release", "upload", args.pmtiles_release, args.listing_files_tiled, "--clobber"])
     Path(args.listing_files_tiled).unlink()
 
+    # Handle force redo bounds file
+    if force_redo_bounds_file:
+        print("Deleting force redo bounds file...")
+        run_command(["gh", "release", "delete-asset", args.gtiffs_release, str(force_redo_bounds_file), '-y'])
+        force_redo_bounds_file.unlink()
+
     # Cleanup
     print("Cleaning up staging directories...")
     shutil.rmtree(tiles_dir, ignore_errors=True)