topo_map_processor 0.1.4__tar.gz → 0.1.5__tar.gz

{topo_map_processor-0.1.4 → topo_map_processor-0.1.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: topo_map_processor
-Version: 0.1.4
+Version: 0.1.5
 Summary: Library for processing topographic maps
 Project-URL: homepage, https://github.com/ramSeraph/topo_map_processor
 Project-URL: repository, https://github.com/ramSeraph/topo_map_processor.git
@@ -172,18 +172,22 @@ I would recommend looking at the [nepal_survey_maps](https://github.com/ramSerap
 
 The `get_intersection_point` method is the most critical part to customize for a new map series. The library provides several helper methods that can be used to implement this logic, each suited for different types of corner features:
 
+![corner for nearest intersection point](/samples/corner3.png)
 -   **`get_nearest_intersection_point`**: This method is useful when the map corners are defined by the intersection of two perpendicular lines (like a simple crosshair). It works by:
     1.  Detecting all horizontal and vertical lines in the corner region.
     2.  Finding all intersection points between these lines.
     3.  Identifying the point closest to the expected corner position, while also matching an expected angle and distance from an anchor point.
 
+![corner for 4way intersection point](/samples/corner3.png)
 -   **`get_4way_intersection_point`**: This is a more robust version for crosshair-style corners. It specifically looks for intersections where four lines meet, making it less prone to errors from stray lines in the map collar.
 
+![corner for biggest contour corner](/samples/corner2.png)
 -   **`get_biggest_contour_corner`**: This strategy is designed for corners that are marked by a solid shape or a filled area (e.g., a colored square or circle). It works by:
     1.  Isolating the specific color of the corner feature.
     2.  Finding the largest contour (shape) of that color in the corner region.
     3.  Returning the outermost point of that contour as the corner.
 
+![corner for nearest intersection point from biggest corner contour](/samples/corner1.png)
 -   **`get_nearest_intersection_point_from_biggest_corner_contour`**: This is a hybrid approach. It first uses the "biggest contour" method to locate a general corner area (e.g., a colored box) and then searches for a precise line intersection *within* that area. This is effective for maps that have both a colored corner block and a fine crosshair inside it.
 
 By combining these methods in your subclass's implementation of `get_intersection_point`, you can build a reliable corner detection system for your specific map type.
@@ -208,46 +212,51 @@ This project provides several command-line tools to work with topographic maps.
 Collects individual GeoJSONL bound files into a single GeoJSON FeatureCollection.
 
 ```bash
-collect-bounds --bounds-dir <directory> --output-file <file>
+collect-bounds --bounds-dir <directory> --output-file <file> [--preexisting-file <file>]
 ```
 
 -   `--bounds-dir`: Directory containing GeoJSONL files. (Required)
 -   `--output-file`: Output GeoJSON file path. (Required)
+-   `--preexisting-file`: Pre existing GeoJSON file path. (Optional)
 
 #### `partition`
 
-Partitions a large set of tiles into smaller PMTiles files. This is useful for managing large datasets, for example, to stay within file size limits for services like GitHub Releases.
+Partitions a large set of tiles into smaller PMTiles files. This is useful for managing large datasets, for example, to stay within file size limits for services like GitHub Releases. When run with `--only-disk`, it creates a new partitioned `pmtiles` fileset from a directory of tiles. When run without, it can be used to re-partition an existing `pmtiles` fileset, for example after having retiled some of the sheets.
 
 ```bash
-partition --to-pmtiles-prefix <prefix> --from-tiles-dir <dir> --name <name> --description <desc> --max-zoom <zoom> [--min-zoom <zoom>] [--from-pmtiles-prefix <prefix>] [--only-disk] (--attribution <text> | --attribution-file <file>)
+partition --to-pmtiles-prefix <prefix> --from-tiles-dir <dir> [--from-pmtiles-prefix <prefix>] [--name <name>] [--description <desc>] [--max-zoom <zoom>] [--min-zoom <zoom>] [--only-disk] (--attribution <text> | --attribution-file <file>)
 ```
 
 -   `--to-pmtiles-prefix`: Prefix for the output PMTiles files. (Required)
 -   `--from-tiles-dir`: Directory containing the input tiles. (Required)
--   `--name`: Name of the mosaic. (Required)
--   `--description`: Description of the mosaic. (Required)
--   `--max-zoom`: Maximum zoom level for the mosaic. (Required)
--   `--min-zoom`: Minimum zoom level for the mosaic (default: 0).
 -   `--from-pmtiles-prefix`: Prefix for input PMTiles files (required if not using `--only-disk`).
+-   `--name`: Name of the mosaic. (Required if using `--only-disk`)
+-   `--description`: Description of the mosaic. (Required if using `--only-disk`)
+-   `--max-zoom`: Maximum zoom level for the mosaic. (Defaults to the max zoom found in the source)
+-   `--min-zoom`: Minimum zoom level for the mosaic (default: 0).
 -   `--only-disk`: Only read tiles from the local disk.
--   `--attribution`: Attribution text for the mosaic. (Required, unless `--attribution-file` is used)
--   `--attribution-file`: File containing attribution text. (Required, unless `--attribution` is used)
+-   `--attribution`: Attribution text for the mosaic. (Required if using `--only-disk`)
+-   `--attribution-file`: File containing attribution text. (Required if using `--only-disk`)
 
 #### `retile`
 
 Retiles a specific list of sheets (GeoTIFFs) and updates the corresponding tiles in an existing tile set. It can determine which other sheets are affected and need to be pulled in to correctly retile the area.
 
+The tool works in two stages. First, it calculates the full set of sheets that need to be processed (the "pull list"), which includes the sheets from the input list plus any adjacent sheets that share tiles. You can run the tool with `--sheets-to-pull-list-outfile` to generate this list and then exit. This is useful for fetching the required GeoTIFFs before proceeding.
+
+In the second stage, it retiles the affected area. It uses the pull list to create a temporary virtual raster (`.vrt`), generates new base tiles for the affected region, and then reconstructs the upper-level tiles by pulling existing, unaffected tiles from the PMTiles source.
+
 ```bash
-retile --retile-list-file <file> --bounds-file <file> --max-zoom <zoom> [--min-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-pmtiles-prefix <prefix>] [--sheets-to-pull-list-outfile <file>]
+retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--min-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-pmtiles-prefix <prefix>] [--sheets-to-pull-list-outfile <file>]
 ```
 
 -   `--retile-list-file`: File containing the list of sheets to retile. (Required)
 -   `--bounds-file`: GeoJSON file containing the list of available sheets and their geographic bounds. (Required)
--   `--max-zoom`: Maximum zoom level to create tiles for. (Required)
--   `--min-zoom`: Minimum zoom level to create tiles for (default: 0).
--   `--tiffs-dir`: Directory where the GeoTIFFs are located.
--   `--tiles-dir`: Directory where the tiles will be created.
--   `--from-pmtiles-prefix`: Prefix for the PMTiles source from which to pull existing tiles.
+-   `--max-zoom`: Maximum zoom level to create tiles for. (Defaults to the max zoom from the PMTiles source)
+-   `--min-zoom`: Minimum zoom level to create tiles for. (Defaults to the min zoom from the PMTiles source)
+-   `--tiffs-dir`: Directory where the GeoTIFFs are located. (Required if not in pull-list generation mode)
+-   `--tiles-dir`: Directory where the tiles will be created. (Required if not in pull-list generation mode)
+-   `--from-pmtiles-prefix`: Prefix for the PMTiles source from which to pull existing tiles. (Required if not in pull-list generation mode)
 -   `--sheets-to-pull-list-outfile`: If provided, the script will only calculate the full list of sheets that need to be processed (including adjacent ones) and write it to this file, then exit.
 
 #### `tile`
@@ -263,18 +272,6 @@ tile <tiles-dir> <tiffs-dir> <max-zoom> [<min-zoom>]
 -   `<max-zoom>`: Maximum zoom level for tiling. (Required)
 -   `<min-zoom>`: Minimum zoom level for tiling (default: 0).
 
-#### `update-bounds`
-
-Updates a main GeoJSON bounds file with new or modified bounds from individual `.geojsonl` files.
-
-```bash
-update-bounds --bounds-file <file> --bounds-dir <dir> --retile-list-file <file>
-```
-
--   `--bounds-file`: The main GeoJSON bounds file to update. (Required)
--   `--bounds-dir`: Directory containing the new or updated individual `.geojsonl` files. (Required)
--   `--retile-list-file`: A file containing the list of sheet names that have been updated (corresponding to the files in `--bounds-dir`). (Required)
-
 ### Running Tools Directly with `uvx`
 
 You can run the command-line tools directly from PyPI without a persistent installation using `uvx`. This is convenient for one-off tasks.

{topo_map_processor-0.1.4 → topo_map_processor-0.1.5}/README.md

@@ -126,18 +126,22 @@ I would recommend looking at the [nepal_survey_maps](https://github.com/ramSerap
 
 The `get_intersection_point` method is the most critical part to customize for a new map series. The library provides several helper methods that can be used to implement this logic, each suited for different types of corner features:
 
+![corner for nearest intersection point](/samples/corner3.png)
 -   **`get_nearest_intersection_point`**: This method is useful when the map corners are defined by the intersection of two perpendicular lines (like a simple crosshair). It works by:
     1.  Detecting all horizontal and vertical lines in the corner region.
     2.  Finding all intersection points between these lines.
     3.  Identifying the point closest to the expected corner position, while also matching an expected angle and distance from an anchor point.
 
+![corner for 4way intersection point](/samples/corner3.png)
 -   **`get_4way_intersection_point`**: This is a more robust version for crosshair-style corners. It specifically looks for intersections where four lines meet, making it less prone to errors from stray lines in the map collar.
 
+![corner for biggest contour corner](/samples/corner2.png)
 -   **`get_biggest_contour_corner`**: This strategy is designed for corners that are marked by a solid shape or a filled area (e.g., a colored square or circle). It works by:
     1.  Isolating the specific color of the corner feature.
     2.  Finding the largest contour (shape) of that color in the corner region.
     3.  Returning the outermost point of that contour as the corner.
 
+![corner for nearest intersection point from biggest corner contour](/samples/corner1.png)
 -   **`get_nearest_intersection_point_from_biggest_corner_contour`**: This is a hybrid approach. It first uses the "biggest contour" method to locate a general corner area (e.g., a colored box) and then searches for a precise line intersection *within* that area. This is effective for maps that have both a colored corner block and a fine crosshair inside it.
 
 By combining these methods in your subclass's implementation of `get_intersection_point`, you can build a reliable corner detection system for your specific map type.
@@ -162,46 +166,51 @@ This project provides several command-line tools to work with topographic maps.
 Collects individual GeoJSONL bound files into a single GeoJSON FeatureCollection.
 
 ```bash
-collect-bounds --bounds-dir <directory> --output-file <file>
+collect-bounds --bounds-dir <directory> --output-file <file> [--preexisting-file <file>]
 ```
 
 -   `--bounds-dir`: Directory containing GeoJSONL files. (Required)
 -   `--output-file`: Output GeoJSON file path. (Required)
+-   `--preexisting-file`: Pre existing GeoJSON file path. (Optional)
 
 #### `partition`
 
-Partitions a large set of tiles into smaller PMTiles files. This is useful for managing large datasets, for example, to stay within file size limits for services like GitHub Releases.
+Partitions a large set of tiles into smaller PMTiles files. This is useful for managing large datasets, for example, to stay within file size limits for services like GitHub Releases. When run with `--only-disk`, it creates a new partitioned `pmtiles` fileset from a directory of tiles. When run without, it can be used to re-partition an existing `pmtiles` fileset, for example after having retiled some of the sheets.
 
 ```bash
-partition --to-pmtiles-prefix <prefix> --from-tiles-dir <dir> --name <name> --description <desc> --max-zoom <zoom> [--min-zoom <zoom>] [--from-pmtiles-prefix <prefix>] [--only-disk] (--attribution <text> | --attribution-file <file>)
+partition --to-pmtiles-prefix <prefix> --from-tiles-dir <dir> [--from-pmtiles-prefix <prefix>] [--name <name>] [--description <desc>] [--max-zoom <zoom>] [--min-zoom <zoom>] [--only-disk] (--attribution <text> | --attribution-file <file>)
 ```
 
 -   `--to-pmtiles-prefix`: Prefix for the output PMTiles files. (Required)
 -   `--from-tiles-dir`: Directory containing the input tiles. (Required)
--   `--name`: Name of the mosaic. (Required)
--   `--description`: Description of the mosaic. (Required)
--   `--max-zoom`: Maximum zoom level for the mosaic. (Required)
--   `--min-zoom`: Minimum zoom level for the mosaic (default: 0).
 -   `--from-pmtiles-prefix`: Prefix for input PMTiles files (required if not using `--only-disk`).
+-   `--name`: Name of the mosaic. (Required if using `--only-disk`)
+-   `--description`: Description of the mosaic. (Required if using `--only-disk`)
+-   `--max-zoom`: Maximum zoom level for the mosaic. (Defaults to the max zoom found in the source)
+-   `--min-zoom`: Minimum zoom level for the mosaic (default: 0).
 -   `--only-disk`: Only read tiles from the local disk.
--   `--attribution`: Attribution text for the mosaic. (Required, unless `--attribution-file` is used)
--   `--attribution-file`: File containing attribution text. (Required, unless `--attribution` is used)
+-   `--attribution`: Attribution text for the mosaic. (Required if using `--only-disk`)
+-   `--attribution-file`: File containing attribution text. (Required if using `--only-disk`)
 
 #### `retile`
 
 Retiles a specific list of sheets (GeoTIFFs) and updates the corresponding tiles in an existing tile set. It can determine which other sheets are affected and need to be pulled in to correctly retile the area.
 
+The tool works in two stages. First, it calculates the full set of sheets that need to be processed (the "pull list"), which includes the sheets from the input list plus any adjacent sheets that share tiles. You can run the tool with `--sheets-to-pull-list-outfile` to generate this list and then exit. This is useful for fetching the required GeoTIFFs before proceeding.
+
+In the second stage, it retiles the affected area. It uses the pull list to create a temporary virtual raster (`.vrt`), generates new base tiles for the affected region, and then reconstructs the upper-level tiles by pulling existing, unaffected tiles from the PMTiles source.
+
 ```bash
-retile --retile-list-file <file> --bounds-file <file> --max-zoom <zoom> [--min-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-pmtiles-prefix <prefix>] [--sheets-to-pull-list-outfile <file>]
+retile --retile-list-file <file> --bounds-file <file> [--max-zoom <zoom>] [--min-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-pmtiles-prefix <prefix>] [--sheets-to-pull-list-outfile <file>]
 ```
 
 -   `--retile-list-file`: File containing the list of sheets to retile. (Required)
 -   `--bounds-file`: GeoJSON file containing the list of available sheets and their geographic bounds. (Required)
--   `--max-zoom`: Maximum zoom level to create tiles for. (Required)
--   `--min-zoom`: Minimum zoom level to create tiles for (default: 0).
--   `--tiffs-dir`: Directory where the GeoTIFFs are located.
--   `--tiles-dir`: Directory where the tiles will be created.
--   `--from-pmtiles-prefix`: Prefix for the PMTiles source from which to pull existing tiles.
+-   `--max-zoom`: Maximum zoom level to create tiles for. (Defaults to the max zoom from the PMTiles source)
+-   `--min-zoom`: Minimum zoom level to create tiles for. (Defaults to the min zoom from the PMTiles source)
+-   `--tiffs-dir`: Directory where the GeoTIFFs are located. (Required if not in pull-list generation mode)
+-   `--tiles-dir`: Directory where the tiles will be created. (Required if not in pull-list generation mode)
+-   `--from-pmtiles-prefix`: Prefix for the PMTiles source from which to pull existing tiles. (Required if not in pull-list generation mode)
 -   `--sheets-to-pull-list-outfile`: If provided, the script will only calculate the full list of sheets that need to be processed (including adjacent ones) and write it to this file, then exit.
 
 #### `tile`
@@ -217,18 +226,6 @@ tile <tiles-dir> <tiffs-dir> <max-zoom> [<min-zoom>]
 -   `<max-zoom>`: Maximum zoom level for tiling. (Required)
 -   `<min-zoom>`: Minimum zoom level for tiling (default: 0).
 
-#### `update-bounds`
-
-Updates a main GeoJSON bounds file with new or modified bounds from individual `.geojsonl` files.
-
-```bash
-update-bounds --bounds-file <file> --bounds-dir <dir> --retile-list-file <file>
-```
-
--   `--bounds-file`: The main GeoJSON bounds file to update. (Required)
--   `--bounds-dir`: Directory containing the new or updated individual `.geojsonl` files. (Required)
--   `--retile-list-file`: A file containing the list of sheet names that have been updated (corresponding to the files in `--bounds-dir`). (Required)
-
 ### Running Tools Directly with `uvx`
 
 You can run the command-line tools directly from PyPI without a persistent installation using `uvx`. This is convenient for one-off tasks.

{topo_map_processor-0.1.4 → topo_map_processor-0.1.5}/pyproject.toml

@@ -31,7 +31,6 @@ repository = "https://github.com/ramSeraph/topo_map_processor.git"
 
 [project.scripts]
 collect-bounds = "topo_map_processor.tools.collect_bounds:cli"
-update-bounds = "topo_map_processor.tools.update_bounds:cli"
 partition = "topo_map_processor.tools.partition:cli"
 retile = "topo_map_processor.tools.retile:cli"
 tile = "topo_map_processor.tools.tile:cli"

{topo_map_processor-0.1.4 → topo_map_processor-0.1.5}/topo_map_processor/_version.py

@@ -17,5 +17,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.1.4'
-__version_tuple__ = version_tuple = (0, 1, 4)
+__version__ = version = '0.1.5'
+__version_tuple__ = version_tuple = (0, 1, 5)

topo_map_processor-0.1.5/topo_map_processor/tools/collect_bounds.py

@@ -0,0 +1,51 @@
+
+import json
+import argparse
+from pathlib import Path
+
+def cli():
+    parser = argparse.ArgumentParser(description='Collect GeoJSONL files into a single GeoJSON file.')
+    parser.add_argument('--bounds-dir', required=True, help='Directory containing GeoJSONL files')
+    parser.add_argument('--output-file', required=True, help='Output GeoJSON file path')
+    parser.add_argument('--preexisting-file', help='preexisting GeoJSON file to base things on', default=None)
+
+    args = parser.parse_args()
+
+    feat_map = {}
+    if args.preexisting_file:
+        preexisting_file = Path(args.preexisting_file)
+        if not preexisting_file.exists():
+            raise FileNotFoundError(f"Preexisting file does not exist: {preexisting_file}")
+
+        existing_data = json.loads(preexisting_file.read_text())
+        for feature in existing_data.get('features', []):
+            if 'properties' in feature and 'id' in feature['properties']:
+                sheet_name = feature['properties']['id']
+                feat_map[sheet_name] = feature
+
+    bounds_dir = Path(args.bounds_dir)
+
+    paths = list(bounds_dir.glob('*.geojsonl'))
+    for p in paths:
+        feat = json.loads(p.read_text())
+        if 'properties' in feat and 'id' in feat['properties']:
+            sheet_name = feat['properties']['id']
+            feat_map[sheet_name] = feat
+
+    with open(args.output_file, 'w') as f:
+        f.write('{ "type": "FeatureCollection", "features": [\n')
+
+        feats = list(feat_map.values())
+        for i,feat in enumerate(feats):
+            line = json.dumps(feat)
+
+            if i != len(feats) - 1:
+                line += ','
+            line += '\n'
+            f.write(line)
+
+        f.write(']}\n')
+
+if __name__ == "__main__":
+    cli()
+

{topo_map_processor-0.1.4 → topo_map_processor-0.1.5}/topo_map_processor/tools/partition.py

@@ -319,11 +319,11 @@ def cli():
     parser.add_argument('--to-pmtiles-prefix', required=True, help='Prefix for the output PMTiles files.')
     parser.add_argument('--from-pmtiles-prefix', help='Prefix for the input PMTiles files. Required if not using --only-disk.')
     parser.add_argument('--from-tiles-dir', required=True, help='Directory containing the input tiles')
-    parser.add_argument('--name', required=True, help='Name of the mosaic.')
-    parser.add_argument('--description', required=True, help='Description of the mosaic.')
+    parser.add_argument('--name', required=False, help='Name of the mosaic.')
+    parser.add_argument('--description', required=False, help='Description of the mosaic.')
     parser.add_argument('--min-zoom', default=0, type=int, help='Minimum zoom level for the mosaic.')
-    parser.add_argument('--max-zoom', required=True, type=int, help='Maximum zoom level for the mosaic.')
-    group = parser.add_mutually_exclusive_group(required=True)
+    parser.add_argument('--max-zoom', required=False, type=int, help='Maximum zoom level for the mosaic.')
+    group = parser.add_mutually_exclusive_group(required=False)
     group.add_argument('--attribution', help='Attribution text for the mosaic.')
     group.add_argument('--attribution-file', help='File containing attribution text for the mosaic.')
     args = parser.parse_args()
@@ -337,19 +337,52 @@ def cli():
     if not args.only_disk and not args.from_pmtiles_prefix:
         parser.error("--from-pmtiles-prefix is required unless --only-disk is set.")
 
+    if args.only_disk:
+        missing_args = []
+        if args.name is None:
+            missing_args.append('--name')
+        if args.description is None:
+            missing_args.append('--description')
+        if attribution_text is None:
+            missing_args.append('--attribution or --attribution-file')
+        if missing_args:
+            parser.error(f'Missing required arguments for --only-disk: {", ".join(missing_args)}')
 
     if args.only_disk:
         reader = DiskSource(args.from_tiles_dir)
     else:
         reader = DiskAndPartitionedPMTilesSource(args.from_tiles_dir, args.from_pmtiles_prefix)
 
+    if args.max_zoom is None:
+        max_zoom = reader.max_zoom
+    else:
+        max_zoom = args.max_zoom
+
+    if args.min_zoom is None:
+        min_zoom = reader.min_zoom
+    else:
+        min_zoom = args.min_zoom
+
+    if args.only_disk:
+        name = args.name
+    else:
+        name = args.name or reader.name
+
+    if args.only_disk:
+        description = args.description
+    else:
+        description = args.description or reader.description
+
+    if not args.only_disk:
+        attribution_text = attribution_text or reader.attribution
+
     print('getting partition info')
     to_partition_file = Path(f'{args.to_pmtiles_prefix}.partition_info.json')
-    partition_info = get_partition_info(reader, args.min_zoom, args.max_zoom, to_partition_file)
+    partition_info = get_partition_info(reader, min_zoom, max_zoom, to_partition_file)
 
     print('creating pmtiles')
     mosaic_data = create_pmtiles(partition_info, reader, args.to_pmtiles_prefix,
-                                 args.name, args.description, attribution_text)
+                                 name, description, attribution_text)
     pprint(mosaic_data)
     if len(mosaic_data) > 1:
         Path(f'{args.to_pmtiles_prefix}.mosaic.json').write_text(json.dumps(mosaic_data))

{topo_map_processor-0.1.4 → topo_map_processor-0.1.5}/topo_map_processor/tools/retile.py

@@ -3,6 +3,7 @@
 # dependencies = [
 #     "mercantile",
 #     "pmtiles",
+#     "shapely",
 # ]
 # ///
 
@@ -18,18 +19,15 @@ from pathlib import Path
 
 
 import mercantile
-from shapely.geometry import mapping
+from shapely.geometry import shape
 
 from osgeo_utils.gdal2tiles import main as gdal2tiles_main
 from osgeo_utils.gdal2tiles import create_overview_tile, TileJobInfo, GDAL2Tiles
 
-
 from tile_sources import PartitionedPMTilesSource, MissingTileError
 
 WEBP_QUALITY = 75
 
-
-
 class AttrDict(dict):
     def __init__(self, *args, **kwargs):
         super(AttrDict, self).__init__(*args, **kwargs)
@@ -110,7 +108,7 @@ def get_tile_file(tile, tiles_dir):
 
 def check_sheets(sheets_to_pull, tiffs_dir):
     for sheet_no in sheets_to_pull:
-        to = tiffs_dir.joinpath(f'{sheet_no}.tif')
+        to = tiffs_dir.joinpath(f'{sheet_no}')
         if not to.exists():
             raise Exception(f'missing file {to}')
 
@@ -122,7 +120,7 @@ def copy_tiles_over(tiles_to_pull, tiles_dir, from_pmtiles_prefix):
         pull_from_pmtiles(to, from_pmtiles_prefix)
 
 
-def create_upper_tiles(tiles_to_create, tiles_dir):
+def create_upper_tiles(z, tiles_to_create, tiles_dir):
     options = AttrDict({
         'resume': True,
         'verbose': False,
@@ -162,7 +160,7 @@ def get_sheet_data(bounds_fname):
     sheets_to_box = {}
     for f in index_data['features']:
         sheet_no = f['properties']['id']
-        geom = mapping(f['geometry'])
+        geom = shape(f['geometry'])
         xmin, ymin, xmax, ymax = geom.bounds
         box = mercantile.LngLatBbox(xmin, ymin, xmax, ymax)
         sheets_to_box[sheet_no] = box
@@ -203,7 +201,7 @@ def create_vrt_file(sheets, tiffs_dir):
     vrt_file = Path(f'{tiffs_dir}/combined.vrt')
     if vrt_file.exists():
         return vrt_file
-    tiff_list = [ tiffs_dir.joinpath(f'{p_sheet}.tif').resolve() for p_sheet in sheets ]
+    tiff_list = [ tiffs_dir.joinpath(f'{p_sheet}').resolve() for p_sheet in sheets ]
     tiff_list = [ str(f) for f in tiff_list if f.exists() ]
 
     tiff_list_str = ' '.join(tiff_list)
@@ -217,8 +215,8 @@ def cli():
     parser = argparse.ArgumentParser()
     parser.add_argument('--retile-list-file', required=True, help='File containing list of sheets to retile')
     parser.add_argument('--bounds-file', required=True, help='Geojson file containing list of available sheets and their georaphic bounds')
-    parser.add_argument('--min-zoom', default=0, type=int, help='Minimum zoom level to create tiles for')
-    parser.add_argument('--max-zoom', required=True, type=int, help='Maximum zoom level to create tiles for')
+    parser.add_argument('--min-zoom', type=int, help='Minimum zoom level to create tiles for')
+    parser.add_argument('--max-zoom', type=int, help='Maximum zoom level to create tiles for')
     parser.add_argument('--sheets-to-pull-list-outfile', default=None,
                         help='Output into which we write the list of sheet that need to be pulled, if set, the script ends after it created the list file')
     parser.add_argument('--from-pmtiles-prefix', help='Prefix to the PMTiles source from which we pull tiles, if needs to be set when --sheets-to-pull-list-outfile is not set')
@@ -239,11 +237,25 @@ def cli():
             parser.error(f"The following arguments are required when --sheets-to-pull-list-outfile is not provided: {', '.join(missing)}")
 
     retile_sheets = Path(args.retile_list_file).read_text().split('\n')
-    retile_sheets = set([ r.strip() for r in retile_sheets if r.strip() != '' ])
+    retile_sheets = set([ r.strip().replace('.tif', '') for r in retile_sheets if r.strip() != '' ])
+
+    if args.from_pmtiles_prefix is None:
+        if not args.max_zoom:
+            parser.error('--max-zoom is required when --from-pmtiles-prefix is not set')
+        max_zoom = args.max_zoom
+
+        if not args.min_zoom:
+            parser.error('--min-zoom is required when --from-pmtiles-prefix is not set')
+        min_zoom = args.min_zoom
+    else:
+        reader = get_pmtiles_reader(args.from_pmtiles_prefix)
+        max_zoom = reader.max_zoom
+        min_zoom = reader.min_zoom
+
 
     print('getting base tiles to sheet mapping')
     sheets_to_box = get_sheet_data(args.bounds_file)
-    sheets_to_base_tiles, base_tiles_to_sheets = get_base_tile_sheet_mappings(sheets_to_box, args.max_zoom)
+    sheets_to_base_tiles, base_tiles_to_sheets = get_base_tile_sheet_mappings(sheets_to_box, max_zoom)
 
     all_affected_tiles = set()
 
@@ -257,7 +269,7 @@ def cli():
     for tile in affected_base_tiles:
         to_add = base_tiles_to_sheets[tile]
         for sheet in to_add:
-            sheets_to_pull.add(sheet)
+            sheets_to_pull.add(sheet + '.tif')
 
     print(f'{sheets_to_pull=}')
         
@@ -268,23 +280,25 @@ def cli():
         exit(0)
 
 
+    tiles_dir = Path(args.tiles_dir)
+    tiffs_dir = Path(args.tiffs_dir)
 
     print('check the sheets availability')
-    check_sheets(sheets_to_pull, args.tiffs_dir)
+    check_sheets(sheets_to_pull, tiffs_dir)
 
     print('creating vrt file from sheets involved')
-    vrt_file = create_vrt_file(sheets_to_pull, args.tiffs_dir)
+    vrt_file = create_vrt_file(sheets_to_pull, tiffs_dir)
 
     Path(args.tiles_dir).mkdir(exist_ok=True, parents=True)
 
     print('creating tiles for base zoom with a vrt')
-    create_base_tiles(f'{vrt_file}', str(args.tiles_dir), f'{args.max_zoom}')
+    create_base_tiles(str(vrt_file), tiles_dir, f'{max_zoom}')
 
     print('deleting unwanted base tiles')
-    delete_unwanted_tiles(affected_base_tiles, args.max_zoom, args.tiles_dir)
+    delete_unwanted_tiles(affected_base_tiles, max_zoom, tiles_dir)
 
     prev_affected_tiles = affected_base_tiles
-    for z in range(args.max_zoom-1, args.min_zoom-1, -1):
+    for z in range(max_zoom-1, min_zoom-1, -1):
         print(f'handling level {z}')
 
         curr_affected_tiles = set()
@@ -299,13 +313,13 @@ def cli():
                     child_tiles_to_pull.add(ctile)
 
         print('copying additional child tiles required for curr level')
-        copy_tiles_over(child_tiles_to_pull, args.tiles_dir, args.from_pmtiles_prefix)
+        copy_tiles_over(child_tiles_to_pull, tiles_dir, args.from_pmtiles_prefix)
 
         print('creating tiles for current level')
-        create_upper_tiles(curr_affected_tiles, args.tiles_dir)
+        create_upper_tiles(z, curr_affected_tiles, tiles_dir)
 
         print('removing unwanted child tiles')
-        delete_unwanted_tiles(prev_affected_tiles, z+1, args.tiles_dir)
+        delete_unwanted_tiles(prev_affected_tiles, z+1, tiles_dir)
 
         prev_affected_tiles = curr_affected_tiles
 

{topo_map_processor-0.1.4 → topo_map_processor-0.1.5}/topo_map_processor/tools/tile_sources.py

@@ -1,6 +1,4 @@
-import os
 import glob
-import json
 
 from pathlib import Path
 
@@ -14,52 +12,87 @@ class MissingTileError(Exception):
 
 class DiskSource:
     def __init__(self, directory):
-        self.dir = directory
+        self.dir = Path(directory)
 
-    def get_tile_from_file(self, fname):
-        parts = fname.split('/')
+    def get_tile_from_file(self, file):
+        parts = file.parts
         tile = mercantile.Tile(z=int(parts[-3]),
                                x=int(parts[-2]),
                                y=int(parts[-1].replace('.webp', '')))
         return tile
 
     def file_from_tile(self, tile):
-        return f'{self.dir}/{tile.z}/{tile.x}/{tile.y}.webp'
+        return self.dir / f'{tile.z}' / f'{tile.x}' / f'{tile.y}.webp'
 
     def get_tile_data(self, tile):
-        fname = self.file_from_tile(tile)
-        try:
-            os.stat(fname)
-        except FileNotFoundError:
+        file = self.file_from_tile(tile)
+        if not file.exists():
             raise MissingTileError()
 
-        with open(fname, 'rb') as f:
-            return f.read()
+        return file.read_bytes()
 
     def get_tile_size(self, tile):
-        fname = self.file_from_tile(tile)
-        try:
-            fstats = os.stat(fname)
-        except FileNotFoundError:
+        file = self.file_from_tile(tile)
+        if not file.exists():
             raise MissingTileError()
-        return fstats.st_size
+
+        return file.stat().st_size
         
     def for_all_z(self, z):
-        fnames = glob.glob(f'{self.dir}/{z}/*/*.webp')
-        for fname in fnames:
-            tile = self.get_tile_from_file(fname)
-            fstats = os.stat(fname)
+        for file in self.dir.glob(f'{z}/*/*.webp'):
+            tile = self.get_tile_from_file(file)
+            fstats = file.stat()
             yield (tile, fstats.st_size)
 
     def all(self):
-        fnames = glob.glob(f'{self.dir}/*/*/*.webp')
-        for fname in fnames:
-            tile = self.get_tile_from_file(fname)
-            t_data = Path(fname).read_bytes()
+        for file in self.dir.glob('*/*/*.webp'):
+            tile = self.get_tile_from_file(file)
+            t_data = file.read_bytes()
             yield (tile, t_data)
 
     def cleanup(self):
         pass
+
+    def get_zomm_levels(self):
+        zoom_levels = []
+        for dir in self.dir.glob('*'):
+            if not dir.is_dir():
+                continue
+
+            try:
+                zoom = int(dir.name)
+            except ValueError:
+                continue
+            zoom_levels.append(zoom)
+        return zoom_levels
+
+    @property
+    def min_zoom(self):
+        zoom_levels = self.get_zomm_levels()
+        if len(zoom_levels) == 0:
+            raise ValueError("No zoom directories found in the disk source.")
+
+        return min(zoom_levels)
+
+    @property
+    def max_zoom(self):
+        zoom_levels = self.get_zomm_levels()
+        if len(zoom_levels) == 0:
+            raise ValueError("No zoom directories found in the disk source.")
+
+        return max(zoom_levels)
+
+    @property
+    def name(self):
+        raise ValueError("DiskSource does not have a name property.")
+        
+    @property
+    def description(self):
+        raise ValueError("DiskSource does not have a description property.")
+
+    @property
+    def attribution(self):
+        raise ValueError("DiskSource does not have a attribution property.")
         
 class PartitionedPMTilesSource:
     def __init__(self, pmtiles_prefix):
@@ -69,13 +102,13 @@ class PartitionedPMTilesSource:
         for fname in pmtiles_files:
             file = open(fname, 'rb')
             self.files.append(file)
-            src = MmapSource(open(fname, 'rb'))
+            src = MmapSource(file)
             reader = PMTilesReader(src)
             self.readers[fname] = reader
 
         self.fnames = list(pmtiles_files)
         fname_to_index = { fname:i for i,fname in enumerate(self.fnames) }
-        self.tiles_to_suffix = {}
+        self.tiles_to_findex = {}
         self.tiles_to_size = {}
         for fname in self.fnames:
             print(f'collecting tile sizes from {fname}')
@@ -86,7 +119,7 @@ class PartitionedPMTilesSource:
                 self.tiles_to_size[tile] = len(t_data)
 
     def get_reader_from_tile(self, tile):
-        if tile not in self.tiles_to_suffix:
+        if tile not in self.tiles_to_findex:
             raise MissingTileError()
         s = self.tiles_to_findex[tile]
         fname = self.fnames[s]
@@ -122,6 +155,34 @@ class PartitionedPMTilesSource:
         for f in self.files:
             f.close()
 
+    @property
+    def min_zoom(self):
+        return min([int(reader.header()['min_zoom']) for reader in self.readers.values()])
+
+    @property
+    def max_zoom(self):
+        return max([int(reader.header()['max_zoom']) for reader in self.readers.values()])
+
+    def get_meta_prop(self, prop_name):
+        for reader in self.readers.values():
+            metadata = reader.metadata()
+            if prop_name in metadata:
+                return metadata[prop_name]
+        raise ValueError(f"Source does not have a {prop_name} property.")
+
+    @property
+    def name(self):
+        return self.get_meta_prop('name')
+        
+    @property
+    def description(self):
+        return self.get_meta_prop('description')
+
+    @property
+    def attribution(self):
+        return self.get_meta_prop('attribution')
+        
+
  
 
 class DiskAndPartitionedPMTilesSource:
@@ -167,4 +228,59 @@ class DiskAndPartitionedPMTilesSource:
         self.dsrc.cleanup()
         self.psrc.cleanup()
 
+    @property
+    def min_zoom(self):
+        min_zooms = []
+
+        try:
+            min_disk_zoom = self.dsrc.min_zoom
+            min_zooms.append(min_disk_zoom)
+        except ValueError:
+            pass
+
+        try:
+            min_pmtiles_zoom = self.psrc.min_zoom
+            min_zooms.append(min_pmtiles_zoom)
+        except ValueError:
+            pass
+
+        if len(min_zooms) == 0:
+            raise ValueError("No zoom directories found in the disk source or PMTiles source.")
+
+        return min(min_zooms)
+
+    @property
+    def max_zoom(self):
+        max_zooms = []
+
+        try:
+            max_disk_zoom = self.dsrc.max_zoom
+            max_zooms.append(max_disk_zoom)
+        except ValueError:
+            pass
+
+        try:
+            max_pmtiles_zoom = self.psrc.max_zoom
+            max_zooms.append(max_pmtiles_zoom)
+        except ValueError:
+            pass
+
+        if len(max_zooms) == 0:
+            raise ValueError("No zoom directories found in the disk source or PMTiles source.")
+
+        return max(max_zooms)
+
+    @property
+    def name(self):
+        return self.psrc.name
+        
+    @property
+    def description(self):
+        return self.psrc.description
+
+    @property
+    def attribution(self):
+        return self.psrc.attribution
+        
+
 

topo_map_processor-0.1.4/topo_map_processor/tools/collect_bounds.py

@@ -1,25 +0,0 @@
-
-import argparse
-
-def cli():
-    parser = argparse.ArgumentParser(description='Collect GeoJSONL files into a single GeoJSON file.')
-    parser.add_argument('--bounds-dir', required=True, help='Directory containing GeoJSONL files')
-    parser.add_argument('--output-file', required=True, help='Output GeoJSON file path')
-
-    args = parser.parse_args()
-
-    paths = list(args.bounds_dir.glob('*.geojsonl'))
-    with open(args.output_file, 'w') as f:
-        f.write('{ "type": "FeatureCollection", "features": [\n')
-        for i,p in enumerate(paths):
-            line = p.read_text().strip('\n')
-
-            if i != len(paths) - 1:
-                line += ','
-            line += '\n'
-            f.write(line)
-        f.write(']}\n')
-
-if __name__ == "__main__":
-    cli()
-

topo_map_processor-0.1.4/topo_map_processor/tools/update_bounds.py

@@ -1,54 +0,0 @@
-
-import json
-import argparse
-from pathlib import Path
-
-def cli():
-    parser = argparse.ArgumentParser(description="Update bounds in a GeoJSON file.")
-    parser.add_argument("--bounds-file", required=True, help="Input GeoJSON file with bounds.")
-    parser.add_argument("--bounds-dir", required=True, help="Directory containing the individual bounds files")
-    parser.add_argument("--retile-list-file", required=True, help="File containing the list of sheets to update.. .tif extension is expected")
-
-    args = parser.parse_args()
-
-    bounds_file = Path(args.bounds_file)
-    bounds_dir = Path(args.bounds_dir)
-    retile_list_file = Path(args.retile_list_file)
-
-    existing_map = {}
-    if not bounds_file.exists():
-        raise FileNotFoundError(f"Bounds file does not exist: {bounds_file}")
-
-    # Read the existing bounds file
-    existing_data = json.loads(bounds_file.read_text())
-    for feature in existing_data.get('features', []):
-        if 'properties' in feature and 'id' in feature['properties']:
-            sheet_name = feature['properties']['id']
-            existing_map[sheet_name] = feature
-
-    sheets = retile_list_file.read_text().strip().splitlines()
-    sheets = [s.strip() for s in sheets if s.strip()]
-    for s in sheets:
-        s = s.replace('.tif', '')
-        bfile = bounds_dir / f'{s}.geojsonl'
-        if not bfile.exists():
-            raise FileNotFoundError(f"Bounds file for sheet '{s}' does not exist: {bfile}")
-
-        feature = json.loads(bfile.read_text())
-        existing_data[s] = feature
-
-
-    all_feats = list(existing_data.values())
-
-    with open(bounds_file, 'w') as f:
-        f.write('{ "type": "FeatureCollection", "features": [\n')
-        for i,feat in enumerate(all_feats):
-            line = json.dumps(feat)
-            if i != len(all_feats) - 1:
-                line += ','
-            line += '\n'
-            f.write(line)
-        f.write(']}\n')
-
-if __name__ == "__main__":
-    cli()