topo_map_processor 0.1.16__tar.gz → 0.1.18__tar.gz

{topo_map_processor-0.1.16 → topo_map_processor-0.1.18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: topo_map_processor
-Version: 0.1.16
+Version: 0.1.18
 Summary: Library for processing topographic maps
 Project-URL: homepage, https://github.com/ramSeraph/topo_map_processor
 Project-URL: repository, https://github.com/ramSeraph/topo_map_processor.git
@@ -249,28 +249,29 @@ This project provides several command-line tools to work with topographic maps.
 Collects individual GeoJSONL bound files into a single GeoJSON FeatureCollection.
 
 ```bash
-collect-bounds --bounds-dir <directory> --output-file <file> [--preexisting-file <file>]
+collect-bounds -b <directory> -o <file> [-u] [-d <id1> <id2> ...]
 ```
 
--   `--bounds-dir`: Directory containing GeoJSONL files. (Required)
--   `--output-file`: Output GeoJSON file path. (Required)
--   `--preexisting-file`: Pre existing GeoJSON file path. (Optional)
+-   `-b`, `--bounds-dir`: Directory containing GeoJSONL files. (Required)
+-   `-o`, `--output-file`: Output GeoJSON file path. (Required)
+-   `-u`, `--update`: Update the output file in place. (Optional)
+-   `-d`, `--delete`: List of ids to delete from the output file (only available in update mode). (Optional)
 
 #### `retile`
 
-Retiles a specific list of sheets (GeoTIFFs) and updates the corresponding tiles in an existing tile set. It can determine which other sheets are affected and need to be pulled in to correctly retile the area.
+Retiles a specific list of sheets (GeoTIFFs) or a specific area and updates the corresponding tiles in an existing tile set. It can determine which other sheets are affected and need to be pulled in to correctly retile the area.
 
 The tool works in two stages. First, it calculates the full set of sheets that need to be processed (the "pull list"), which includes the sheets from the input list plus any adjacent sheets that share tiles. You can run the tool with `--sheets-to-pull-list-outfile` to generate this list and then exit. This is useful for fetching the required GeoTIFFs before proceeding.
 
 In the second stage, it retiles the affected area. It uses the pull list to create a temporary virtual raster (`.vrt`), generates new base tiles for the affected region, and then reconstructs the upper-level tiles by pulling existing, unaffected tiles from the PMTiles source.
 
 ```bash
-retile --retile-list-file <file> --bounds-file <file> [--dummy-bounds-file <file>] [--max-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-source <source>] [--sheets-to-pull-list-outfile <file>] [--num-parallel <processes>] [--tile-quality <quality>]
+retile [--retile-list-file <file>] --bounds-file <file> [--force-redo-bounds-file <file>] [--max-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-source <source>] [--sheets-to-pull-list-outfile <file>] [--num-parallel <processes>] [--tile-quality <quality>]
 ```
 
--   `--retile-list-file`: File containing the list of sheets to retile. (Required)
+-   `--retile-list-file`: File containing the list of sheets to retile. (Optional)
 -   `--bounds-file`: GeoJSON file containing the list of available sheets and their geographic bounds. (Required)
--   `--dummy-bounds-file`: GeoJSON file containing geographic bounds of areas that need to be updated. These areas might not have any new sheets. (Optional)
+-   `--force-redo-bounds-file`: GeoJSON file containing geographic bounds of areas that need to be updated. These areas might not have any new sheets. (Optional)
 -   `--max-zoom`: Maximum zoom level to create tiles for. Defaults to the max zoom from the source if `--from-source` is provided, otherwise it is required.
 -   `--tiffs-dir`: Directory where the GeoTIFFs are located. (Required if not in pull-list generation mode)
 -   `--tiles-dir`: Directory where the tiles will be created. (Required if not in pull-list generation mode)
@@ -279,6 +280,18 @@ retile --retile-list-file <file> --bounds-file <file> [--dummy-bounds-file <file
 -   `--num-parallel`: Number of parallel processes to use for tiling (default: number of CPU cores).
 -   `--tile-quality`: Quality of compression for webp and jpeg (default: 75).
 
+#### `create-force-redo-bounds`
+
+Creates a GeoJSON file that defines areas to be forcibly re-tiled, based on the differences between an old and a new bounds file. This is useful for identifying regions where map sheets may have been updated, moved, or removed.
+
+```bash
+create-force-redo-bounds --old-bounds-file <file> --new-bounds-file <file> --output-file <file>
+```
+
+-   `--old-bounds-file`: The previous version of the bounds GeoJSON file. (Required)
+-   `--new-bounds-file`: The new version of the bounds GeoJSON file. (Required)
+-   `--output-file`: The path to write the output GeoJSON file, which will contain the bounds of the areas to be force-retiled. (Required)
+
 #### `tile`
 
 Creates web map tiles from a directory of GeoTIFF files. It combines them into a virtual raster (`.vrt`) for efficient processing.
@@ -316,6 +329,49 @@ GDAL_VERSION=$(gdalinfo --version | cut -d"," -f1 | cut -d" " -f2)
 uvx --with numpy --with pillow --with gdal==$GDAL_VERSION --from topo_map_processor tile --help
 ```
 
+### Map Update Workflow
+
+This project provides a robust workflow for updating an existing tile set when map sheets are added, removed, or modified. The primary tools for this are `retile` and `create-force-redo-bounds`.
+
+The general workflow is as follows:
+
+1.  **Generate New Bounds**: After processing new or updated map scans with the library, you will have a new set of COGs and `.geojsonl` bound files. Use the `collect-bounds` tool to create an updated `bounds.geojson` file that reflects the current state of your map collection.
+
+2.  **Identify Changes**: Use the `create-force-redo-bounds` tool to compare your old `bounds.geojson` with the new one.
+
+    ```bash
+    create-force-redo-bounds --old-bounds-file old-bounds.geojson --new-bounds-file new-bounds.geojson --output-file force-redo.geojson
+    ```
+
+    This command analyzes the differences:
+    *   **Added Sheets**: If a map sheet exists in `new-bounds.geojson` but not in `old-bounds.geojson`.
+    *   **Deleted Sheets**: If a sheet is in `old-bounds.geojson` but has been removed from `new-bounds.geojson`.
+    *   **Modified Sheets**: If a sheet's geometry (boundary) or its content (indicated by a change in the `digest` property) has been updated.
+
+    The output, `force-redo.geojson`, will contain the geographic areas for all these changes, which tells the `retile` tool exactly which regions of the map need to be regenerated.
+
+3.  **Retile Affected Areas**: Use the `retile` tool to perform the update. The `force-redo.geojson` file you created contains all the information about what has changed, so a separate list of sheets is not required.
+
+    ```bash
+    retile --bounds-file new-bounds.geojson \
+           --force-redo-bounds-file force-redo.geojson \
+           --from-source "existing-tiles/*.pmtiles" \
+           --tiffs-dir /path/to/cogs \
+           --tiles-dir /path/to/output/tiles
+    ```
+
+    The `retile` tool will:
+    *   Identify all GeoTIFFs needed to regenerate the affected tiles (including adjacent, unchanged sheets).
+    *   Generate new base tiles for the changed areas.
+    *   Copy unchanged tiles from your existing tile set (`--from-source`).
+    *   Re-create parent tiles up the zoom levels.
+
+This workflow ensures that your tile set is updated efficiently, without having to re-process the entire map collection.
+
+## Alternate Tools
+
+For a more saner/cleaner version of doing the georeferencing/tiling/retiling without using GDAL check out [wladich/map-tiler](https://github.com/wladich/map-tiler)
+
 ## Contributing
 
 Contributions are welcome! Please feel free to submit a pull request.

{topo_map_processor-0.1.16 → topo_map_processor-0.1.18}/README.md

@@ -174,28 +174,29 @@ This project provides several command-line tools to work with topographic maps.
 Collects individual GeoJSONL bound files into a single GeoJSON FeatureCollection.
 
 ```bash
-collect-bounds --bounds-dir <directory> --output-file <file> [--preexisting-file <file>]
+collect-bounds -b <directory> -o <file> [-u] [-d <id1> <id2> ...]
 ```
 
--   `--bounds-dir`: Directory containing GeoJSONL files. (Required)
--   `--output-file`: Output GeoJSON file path. (Required)
--   `--preexisting-file`: Pre existing GeoJSON file path. (Optional)
+-   `-b`, `--bounds-dir`: Directory containing GeoJSONL files. (Required)
+-   `-o`, `--output-file`: Output GeoJSON file path. (Required)
+-   `-u`, `--update`: Update the output file in place. (Optional)
+-   `-d`, `--delete`: List of ids to delete from the output file (only available in update mode). (Optional)
 
 #### `retile`
 
-Retiles a specific list of sheets (GeoTIFFs) and updates the corresponding tiles in an existing tile set. It can determine which other sheets are affected and need to be pulled in to correctly retile the area.
+Retiles a specific list of sheets (GeoTIFFs) or a specific area and updates the corresponding tiles in an existing tile set. It can determine which other sheets are affected and need to be pulled in to correctly retile the area.
 
 The tool works in two stages. First, it calculates the full set of sheets that need to be processed (the "pull list"), which includes the sheets from the input list plus any adjacent sheets that share tiles. You can run the tool with `--sheets-to-pull-list-outfile` to generate this list and then exit. This is useful for fetching the required GeoTIFFs before proceeding.
 
 In the second stage, it retiles the affected area. It uses the pull list to create a temporary virtual raster (`.vrt`), generates new base tiles for the affected region, and then reconstructs the upper-level tiles by pulling existing, unaffected tiles from the PMTiles source.
 
 ```bash
-retile --retile-list-file <file> --bounds-file <file> [--dummy-bounds-file <file>] [--max-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-source <source>] [--sheets-to-pull-list-outfile <file>] [--num-parallel <processes>] [--tile-quality <quality>]
+retile [--retile-list-file <file>] --bounds-file <file> [--force-redo-bounds-file <file>] [--max-zoom <zoom>] [--tiffs-dir <dir>] [--tiles-dir <dir>] [--from-source <source>] [--sheets-to-pull-list-outfile <file>] [--num-parallel <processes>] [--tile-quality <quality>]
 ```
 
--   `--retile-list-file`: File containing the list of sheets to retile. (Required)
+-   `--retile-list-file`: File containing the list of sheets to retile. (Optional)
 -   `--bounds-file`: GeoJSON file containing the list of available sheets and their geographic bounds. (Required)
--   `--dummy-bounds-file`: GeoJSON file containing geographic bounds of areas that need to be updated. These areas might not have any new sheets. (Optional)
+-   `--force-redo-bounds-file`: GeoJSON file containing geographic bounds of areas that need to be updated. These areas might not have any new sheets. (Optional)
 -   `--max-zoom`: Maximum zoom level to create tiles for. Defaults to the max zoom from the source if `--from-source` is provided, otherwise it is required.
 -   `--tiffs-dir`: Directory where the GeoTIFFs are located. (Required if not in pull-list generation mode)
 -   `--tiles-dir`: Directory where the tiles will be created. (Required if not in pull-list generation mode)
@@ -204,6 +205,18 @@ retile --retile-list-file <file> --bounds-file <file> [--dummy-bounds-file <file
 -   `--num-parallel`: Number of parallel processes to use for tiling (default: number of CPU cores).
 -   `--tile-quality`: Quality of compression for webp and jpeg (default: 75).
 
+#### `create-force-redo-bounds`
+
+Creates a GeoJSON file that defines areas to be forcibly re-tiled, based on the differences between an old and a new bounds file. This is useful for identifying regions where map sheets may have been updated, moved, or removed.
+
+```bash
+create-force-redo-bounds --old-bounds-file <file> --new-bounds-file <file> --output-file <file>
+```
+
+-   `--old-bounds-file`: The previous version of the bounds GeoJSON file. (Required)
+-   `--new-bounds-file`: The new version of the bounds GeoJSON file. (Required)
+-   `--output-file`: The path to write the output GeoJSON file, which will contain the bounds of the areas to be force-retiled. (Required)
+
 #### `tile`
 
 Creates web map tiles from a directory of GeoTIFF files. It combines them into a virtual raster (`.vrt`) for efficient processing.
@@ -241,6 +254,49 @@ GDAL_VERSION=$(gdalinfo --version | cut -d"," -f1 | cut -d" " -f2)
 uvx --with numpy --with pillow --with gdal==$GDAL_VERSION --from topo_map_processor tile --help
 ```
 
+### Map Update Workflow
+
+This project provides a robust workflow for updating an existing tile set when map sheets are added, removed, or modified. The primary tools for this are `retile` and `create-force-redo-bounds`.
+
+The general workflow is as follows:
+
+1.  **Generate New Bounds**: After processing new or updated map scans with the library, you will have a new set of COGs and `.geojsonl` bound files. Use the `collect-bounds` tool to create an updated `bounds.geojson` file that reflects the current state of your map collection.
+
+2.  **Identify Changes**: Use the `create-force-redo-bounds` tool to compare your old `bounds.geojson` with the new one.
+
+    ```bash
+    create-force-redo-bounds --old-bounds-file old-bounds.geojson --new-bounds-file new-bounds.geojson --output-file force-redo.geojson
+    ```
+
+    This command analyzes the differences:
+    *   **Added Sheets**: If a map sheet exists in `new-bounds.geojson` but not in `old-bounds.geojson`.
+    *   **Deleted Sheets**: If a sheet is in `old-bounds.geojson` but has been removed from `new-bounds.geojson`.
+    *   **Modified Sheets**: If a sheet's geometry (boundary) or its content (indicated by a change in the `digest` property) has been updated.
+
+    The output, `force-redo.geojson`, will contain the geographic areas for all these changes, which tells the `retile` tool exactly which regions of the map need to be regenerated.
+
+3.  **Retile Affected Areas**: Use the `retile` tool to perform the update. The `force-redo.geojson` file you created contains all the information about what has changed, so a separate list of sheets is not required.
+
+    ```bash
+    retile --bounds-file new-bounds.geojson \
+           --force-redo-bounds-file force-redo.geojson \
+           --from-source "existing-tiles/*.pmtiles" \
+           --tiffs-dir /path/to/cogs \
+           --tiles-dir /path/to/output/tiles
+    ```
+
+    The `retile` tool will:
+    *   Identify all GeoTIFFs needed to regenerate the affected tiles (including adjacent, unchanged sheets).
+    *   Generate new base tiles for the changed areas.
+    *   Copy unchanged tiles from your existing tile set (`--from-source`).
+    *   Re-create parent tiles up the zoom levels.
+
+This workflow ensures that your tile set is updated efficiently, without having to re-process the entire map collection.
+
+## Alternate Tools
+
+For a more saner/cleaner version of doing the georeferencing/tiling/retiling without using GDAL check out [wladich/map-tiler](https://github.com/wladich/map-tiler)
+
 ## Contributing
 
 Contributions are welcome! Please feel free to submit a pull request.

{topo_map_processor-0.1.16 → topo_map_processor-0.1.18}/pyproject.toml

@@ -68,6 +68,7 @@ collect-bounds = "topo_map_processor.tools.collect_bounds:cli"
 retile = "topo_map_processor.tools.retile:cli"
 tile = "topo_map_processor.tools.tile:cli"
 retile-e2e = "topo_map_processor.tools.retile_e2e:cli"
+create-force-redo-bounds = "topo_map_processor.tools.create_force_redo_bounds:cli"
 
 [build-system]
 requires = [

{topo_map_processor-0.1.16 → topo_map_processor-0.1.18}/topo_map_processor/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.1.16'
-__version_tuple__ = version_tuple = (0, 1, 16)
+__version__ = version = '0.1.18'
+__version_tuple__ = version_tuple = (0, 1, 18)
 
 __commit_id__ = commit_id = None

{topo_map_processor-0.1.16 → topo_map_processor-0.1.18}/topo_map_processor/processor.py

@@ -4,6 +4,7 @@ import time
 import json
 import pickle
 import shutil
+import hashlib
 import subprocess
 
 from pathlib import Path
@@ -81,6 +82,7 @@ class TopoMapProcessor:
         self.small_img = None
         self.mapbox_corners = None
         self.rotation_reversal_params = None
+        self.digest = None
 
         self.extents = extra.get('extents', None)
         self.pixel_cutlines = extra.get('pixel_cutlines', [])
@@ -106,6 +108,7 @@ class TopoMapProcessor:
         self.use_bbox_area = extra.get('use_bbox_area', True)
         self.shrunk_map_area_corners = extra.get('shrunk_map_area_corners', None)
         self.poly_approx_factor = extra.get('poly_approx_factor', 0.001)
+        self.map_area_ratio_thresh = extra.get('map_area_ratio_thresh', 0.5)
 
         # corners related
         self.corner_ratio = extra.get('corner_ratio', 400.0 / 9000.0)
@@ -132,6 +135,7 @@ class TopoMapProcessor:
         self.warp_jpeg_export_quality = extra.get('warp_jpeg_export_quality', 75)
         self.warp_output_height = extra.get('warp_output_height', 6500)
 
+
         self.color_map = {
             'black':   ((0, 0, 0), (179, 255, 130)),
             'greyish': ((0, 0, 50), (179, 150, 192)),
@@ -484,6 +488,10 @@ class TopoMapProcessor:
         ls = LineString(line)
         line_poly = ls.buffer(line_buf, resolution=1, cap_style=CAP_STYLE.flat).intersection(limits)
         blur_poly = ls.buffer(blur_buf, resolution=1, cap_style=CAP_STYLE.flat).intersection(limits)
+        print(f'removing line {line} with line_buf {line_buf}, blur_buf {blur_buf}, blur_kern {blur_kern}, blur_repeat {blur_repeat}')
+        if blur_poly.is_empty:
+            print('blur poly empty, nothing to do')
+            return
         bb = blur_poly.bounds
         bb = [ round(x) for x in bb ]
         # restrict to a small img strip to make things less costly
@@ -810,7 +818,7 @@ class TopoMapProcessor:
 
                 h, w = img.shape[:2]
                 total_area = w * h
-                if total_area / map_area > 2:
+                if map_area / total_area < self.map_area_ratio_thresh:
                     raise Exception(f'map area less than expected, {map_area=}, {total_area=}')
     
                 self.show_contours(img_mask, [map_contour])
@@ -1088,7 +1096,8 @@ class TopoMapProcessor:
         if remove_text:
             self.remove_text(img_mask)
 
-        self.remove_corner_edges(img_mask, direction, remove_corner_edges_ratio)
+        if remove_corner_edges_ratio is not None:
+            self.remove_corner_edges(img_mask, direction, remove_corner_edges_ratio)
 
         h, w = img_mask.shape[:2]
         self.show_image(img_mask)
@@ -1166,7 +1175,8 @@ class TopoMapProcessor:
 
         anchor_corner = (ax, ay)
 
-        self.remove_corner_edges(img_mask, direction, remove_corner_edges_ratio)
+        if remove_corner_edges_ratio is not None:
+            self.remove_corner_edges(img_mask, direction, remove_corner_edges_ratio)
 
         frame_contour = self.get_biggest_contour(img_mask, corner_erode, True, max_corner_contour_area_ratio)
 
@@ -1230,7 +1240,8 @@ class TopoMapProcessor:
 
         img_c_mask = self.get_color_mask(img, corner_contour_color)
         self.remove_text(img_c_mask)
-        self.remove_corner_edges(img_c_mask, direction, self.remove_corner_edges_ratio)
+        if remove_corner_edges_ratio is not None:
+            self.remove_corner_edges(img_c_mask, direction, remove_corner_edges_ratio)
         frame_contour = self.get_biggest_contour(img_c_mask, self.corner_erode, True, max_corner_contour_area_ratio)
 
         frame_contour_area = self.get_bbox_area_simple(frame_contour)
@@ -1250,6 +1261,7 @@ class TopoMapProcessor:
             fx + fw * ( 1 if direction[0] < 0 else 0), 
             fy + fh * ( 1 if direction[1] < 0 else 0), 
         ]
+
         frame_diag_len = self.get_distance((0,0), (fh,fw))
         self.show_points([frame_anchor_point], img_c_mask, [0,0,255])
 
@@ -1294,10 +1306,24 @@ class TopoMapProcessor:
         map_poly_points = self.reorder_poly_points(map_poly_points)
         print(f'{map_poly_points=}')
 
+        all_corner_overrides_given = all([ c is not None for c in corner_overrides ])
+        if all_corner_overrides_given:
+            return corner_overrides
+
+        some_corner_overrides_given = any([ c is not None for c in corner_overrides ])
+        if some_corner_overrides_given and  len(corner_overrides) != len(map_poly_points):
+            raise ValueError("If some corner overrides are given, either all must be given or the number of overrides must match the number of polygon points.")
+
         #num_corners = len(map_poly_points)
         box = LinearRing(map_poly_points + [map_poly_points[0]])
         buffered_poly = box.buffer(-cw/2, single_sided=True, join_style=JOIN_STYLE.mitre)
-        inner_ring = buffered_poly.interiors[0]
+
+        # TODO: this is a hack.. understand why this happens and fix it properly
+        if len(buffered_poly.interiors) == 0:
+            inner_ring = buffered_poly.exterior
+        else:
+            inner_ring = buffered_poly.interiors[0]
+
         corner_centers = list(inner_ring.coords)
         corner_centers.reverse()
         print(f'{corner_centers=}')
@@ -1390,7 +1416,7 @@ class TopoMapProcessor:
         corner_overrides = [ (c[0] - map_bbox_scaled[0], c[1] - map_bbox_scaled[1]) if c is not None else None for c in corner_overrides_full ]
         map_poly_points_scaled = [ (p[0] - map_bbox_scaled[0], p[1] - map_bbox_scaled[1]) for p in map_poly_points_scaled ]
 
-        if self.extents is None:
+        if len(corner_overrides) == 4:
             corners = self.locate_corners(map_img, corner_overrides)
         else:
             corners = self.locate_corners_generic(map_img, map_poly_points_scaled, corner_overrides)
@@ -1610,6 +1636,21 @@ class TopoMapProcessor:
         translate_cmd = f'gdal_translate {creation_options} {perf_options} {gcp_str} -a_srs "{crs_proj}" -of GTiff {str(from_file)} {str(georef_file)}' 
         self.run_external(translate_cmd)
 
+    def get_digest(self):
+        if self.digest is not None:
+            return self.digest
+
+        hasher = hashlib.sha256()
+        export_file = self.get_export_file()
+        with open(export_file, 'rb') as f:
+            while True:
+                data = f.read(65536)
+                if not data:
+                    break
+                hasher.update(data)
+        self.digest = hasher.hexdigest()
+        return self.digest
+
     def get_cutline_props(self):
         crs_proj = self.get_crs_proj()
 
@@ -1687,7 +1728,7 @@ class TopoMapProcessor:
         workdir = self.get_workdir()
         rotated_info_file = workdir.joinpath('rotated_info.txt')
         if not rotated_info_file.exists():
-            raise Exception(f'rotated_info.txt does not exist in {workdir}')
+            return RotationReversalParams(False, 0.0, None, None)
 
         parts = rotated_info_file.read_text().strip().split(',')
         angle = float(parts[0])
@@ -1813,6 +1854,32 @@ class TopoMapProcessor:
 
         self.run_external(f'ogr2ogr -t_srs EPSG:4326 -s_srs "{crs_proj}" -f GeoJSONSeq {str(bounds_file)} {cutline_file}')
 
+    def add_digest_to_bounds_file(self):
+        bounds_dir = self.get_bounds_dir()
+
+        bounds_file = bounds_dir.joinpath(f'{self.get_id()}.geojsonl')
+        if not bounds_file.exists():
+            print(f'{bounds_file} does not exist.. cannot add digest')
+            return
+
+        digest = self.get_digest()
+
+        with open(bounds_file, 'r') as f:
+            lines = f.readlines()
+
+        if len(lines) != 1:
+            print(f'expected 1 line in {bounds_file}, got {len(lines)}')
+            return
+
+        feature = json.loads(lines[0])
+        if 'properties' not in feature:
+            feature['properties'] = {}
+
+        feature['properties']['digest'] = digest
+
+        with open(bounds_file, 'w') as f:
+            f.write(json.dumps(feature) + '\n')
+
     def warp(self):
         workdir = self.get_workdir()
 
@@ -1851,6 +1918,9 @@ class TopoMapProcessor:
 
         self.export_gtiff(str(final_file), str(export_file), self.jpeg_export_quality)
 
+        self.add_digest_to_bounds_file()
+
+
     def remove_insets(self):
         if self.inset_pixel_cutlines is None or len(self.inset_pixel_cutlines) == 0:
             return

{topo_map_processor-0.1.16 → topo_map_processor-0.1.18}/topo_map_processor/tools/collect_bounds.py

@@ -5,23 +5,32 @@ from pathlib import Path
 
 def cli():
     parser = argparse.ArgumentParser(description='Collect GeoJSONL files into a single GeoJSON file.')
-    parser.add_argument('--bounds-dir', required=True, help='Directory containing GeoJSONL files')
-    parser.add_argument('--output-file', required=True, help='Output GeoJSON file path')
-    parser.add_argument('--preexisting-file', help='preexisting GeoJSON file to base things on', default=None)
+    parser.add_argument('-b', '--bounds-dir', required=True, help='Directory containing GeoJSONL files')
+    parser.add_argument('-o', '--output-file', required=True, help='Output GeoJSON file path')
+    parser.add_argument('-u', '--update', action='store_true', help='Update the output file in place')
+    parser.add_argument('-d', '--delete', nargs='+', help='List of ids to delete from the output file (only in update mode)')
 
     args = parser.parse_args()
 
+    if args.delete and not args.update:
+        raise parser.error("--delete can only be used with --update")
+
     feat_map = {}
-    if args.preexisting_file:
-        preexisting_file = Path(args.preexisting_file)
-        if not preexisting_file.exists():
-            raise FileNotFoundError(f"Preexisting file does not exist: {preexisting_file}")
+    if args.update:
+        output_file = Path(args.output_file)
+        if not output_file.exists():
+            raise FileNotFoundError(f"Output file does not exist for updating: {output_file}")
 
-        existing_data = json.loads(preexisting_file.read_text())
+        existing_data = json.loads(output_file.read_text())
         for feature in existing_data.get('features', []):
             if 'properties' in feature and 'id' in feature['properties']:
                 sheet_name = feature['properties']['id']
                 feat_map[sheet_name] = feature
+    
+    if args.delete:
+        for sheet_id in args.delete:
+            if sheet_id in feat_map:
+                del feat_map[sheet_id]
 
     bounds_dir = Path(args.bounds_dir)
 

topo_map_processor-0.1.18/topo_map_processor/tools/create_force_redo_bounds.py

@@ -0,0 +1,100 @@
+import argparse
+import json
+import sys
+from pathlib import Path
+
+def get_sheet_map(bounds_file):
+    """Reads a bounds file and returns a map of sheet ID to feature."""
+    with open(bounds_file, 'r') as f:
+        data = json.load(f)
+    
+    sheet_map = {}
+    for feature in data.get('features', []):
+        if 'properties' in feature and 'id' in feature['properties']:
+            sheet_id = feature['properties']['id']
+            sheet_map[sheet_id] = feature
+    return sheet_map
+
+def main(argv):
+    parser = argparse.ArgumentParser(description='Create a force redo bounds file by comparing old and new bounds.')
+    parser.add_argument('--old-bounds-file', required=True, help='Path to the old GeoJSON bounds file.')
+    parser.add_argument('--new-bounds-file', required=True, help='Path to the new GeoJSON bounds file.')
+    parser.add_argument('--output-file', required=True, help='Path to write the output GeoJSON file with force redo bounds.')
+
+    args = parser.parse_args(argv)
+
+    old_sheets = get_sheet_map(args.old_bounds_file)
+    new_sheets = get_sheet_map(args.new_bounds_file)
+
+    old_ids = set(old_sheets.keys())
+    new_ids = set(new_sheets.keys())
+
+    added_ids = new_ids - old_ids
+    deleted_ids = old_ids - new_ids
+    common_ids = old_ids.intersection(new_ids)
+
+    output_features = []
+
+    # Added sheets
+    for sheet_id in added_ids:
+        feature = new_sheets[sheet_id]
+        output_features.append({
+            "type": "Feature",
+            "geometry": feature['geometry'],
+            "properties": { "reason": f"Sheet {sheet_id} added" }
+        })
+
+    # Deleted sheets
+    for sheet_id in deleted_ids:
+        feature = old_sheets[sheet_id]
+        output_features.append({
+            "type": "Feature",
+            "geometry": feature['geometry'],
+            "properties": { "reason": f"Sheet {sheet_id} deleted" }
+        })
+
+    # Common sheets (check for modifications)
+    for sheet_id in common_ids:
+        old_feature = old_sheets[sheet_id]
+        new_feature = new_sheets[sheet_id]
+        old_geom = old_feature.get('geometry')
+        new_geom = new_feature.get('geometry')
+        geom_changed = json.dumps(old_geom, sort_keys=True) != json.dumps(new_geom, sort_keys=True)
+        old_digest = old_feature.get('properties', {}).get('digest')
+        new_digest = new_feature.get('properties', {}).get('digest')
+        digest_changed = old_digest != new_digest
+        if geom_changed:
+            output_features.append({
+                "type": "Feature",
+                "geometry": old_geom,
+                "properties": { "reason": f"Sheet {sheet_id} geometry changed (old)" }
+            })
+            output_features.append({
+                "type": "Feature",
+                "geometry": new_geom,
+                "properties": { "reason": f"Sheet {sheet_id} geometry changed (new)" }
+            })
+        elif digest_changed:
+            output_features.append({
+                "type": "Feature",
+                "geometry": new_geom,
+                "properties": { "reason": f"Sheet {sheet_id} digest changed" }
+            })
+
+    with open(args.output_file, 'w') as f:
+        f.write('{ "type": "FeatureCollection", "features": [\n')
+        for i, feature in enumerate(output_features):
+            line = json.dumps(feature)
+            if i < len(output_features) - 1:
+                line += ','
+            line += '\n'
+            f.write(line)
+        f.write(']}\n')
+
+    print(f"Created force redo bounds file at {args.output_file} with {len(output_features)} features.")
+
+def cli():
+    main(sys.argv[1:])
+
+if __name__ == "__main__":
+    cli()

{topo_map_processor-0.1.16 → topo_map_processor-0.1.18}/topo_map_processor/tools/retile.py

@@ -364,8 +364,10 @@ def get_base_tile_sheet_mappings(sheets_to_box, base_zoom):
 
 def assess_sheet_requirements(retile_list_file, bounds_file, force_redo_bounds_file, max_zoom):
 
-    retile_sheets = retile_list_file.read_text().split('\n')
-    retile_sheets = set([ r.strip().replace('.tif', '') for r in retile_sheets if r.strip() != '' ])
+    retile_sheets = set()
+    if retile_list_file:
+        retile_sheets = retile_list_file.read_text().split('\n')
+        retile_sheets = set([ r.strip().replace('.tif', '') for r in retile_sheets if r.strip() != '' ])
 
     print('getting base tiles to sheet mapping')
     sheets_to_box = get_sheet_data(bounds_file)
@@ -414,7 +416,7 @@ def create_dummy_sheets(dummy_bounds_file, tiffs_dir):
 def retile_main(args):
 
     parser = argparse.ArgumentParser()
-    parser.add_argument('--retile-list-file', required=True, help='File containing list of sheets to retile')
+    parser.add_argument('--retile-list-file', required=False, default=None, help='File containing list of sheets to retile')
     parser.add_argument('--bounds-file', required=True, help='Geojson file containing list of available sheets and their georaphic bounds')
     parser.add_argument('--force-redo-bounds-file', required=False, help='Geojson file containing georaphic bounds of areas that need to be updated, these areas might not have any new sheets')
     parser.add_argument('--max-zoom', type=int, help='Maximum zoom level to create tiles for, needed only when --from-source is not provided')
@@ -428,8 +430,11 @@ def retile_main(args):
 
     args = parser.parse_args(args)
 
-    retile_list_file = Path(args.retile_list_file)
-    if not retile_list_file.exists():
+    if not args.retile_list_file and not args.force_redo_bounds_file:
+        parser.error('At least one of --retile-list-file or --force-redo-bounds-file must be provided.')
+
+    retile_list_file = Path(args.retile_list_file) if args.retile_list_file else None
+    if retile_list_file and not retile_list_file.exists():
         parser.error(f'Retile list file {args.retile_list_file} does not exist')
 
     bounds_file = Path(args.bounds_file)

{topo_map_processor-0.1.16 → topo_map_processor-0.1.18}/topo_map_processor/tools/retile_e2e.py

@@ -1,7 +1,6 @@
-
 import argparse
-import csv
 import glob
+import json
 import shutil
 import subprocess
 import sys
@@ -11,6 +10,7 @@ from multiprocessing import set_start_method
 from pmtiles_mosaic.partition import partition_main
 
 from .retile import retile_main
+from .create_force_redo_bounds import main as create_force_redo_bounds_main
 
 
 def run_command(cmd, check=True, cwd=None):
@@ -50,7 +50,7 @@ def cli():
     parser.add_argument("-p", "--pmtiles-release", required=True, help="PMTiles release tag")
     parser.add_argument("-g", "--gtiffs-release", required=True, help="GeoTIFFs release tag")
     parser.add_argument("-x", "--pmtiles-prefix", required=True, help="Prefix for PMTiles files")
-    parser.add_argument("-l", "--listing-files-tiled", required=True, help="Name of the tiled listing file")
+    parser.add_argument("--bounds-file-tiled", required=True, help="Name of the bounds file in the pmtiles release (the old bounds file)")
     args = parser.parse_args()
 
     # Define paths and filenames
@@ -59,7 +59,10 @@ def cli():
     tiffs_dir = Path("staging/gtiffs/")
     from_pmtiles_dir = Path("staging/pmtiles")
     to_pmtiles_dir = Path("export/pmtiles")
-    retile_list_file = Path("to_retile.txt")
+    
+    old_bounds_file = Path("old_bounds.geojson")
+    new_bounds_file = Path("new_bounds.geojson")
+    force_redo_bounds_file = Path("force_redo_bounds.geojson")
 
     from_pmtiles_prefix = from_pmtiles_dir / args.pmtiles_prefix
     to_pmtiles_prefix = to_pmtiles_dir / args.pmtiles_prefix
@@ -68,90 +71,71 @@ def cli():
     for d in [tiles_dir, tiffs_dir, from_pmtiles_dir, to_pmtiles_dir]:
         d.mkdir(parents=True, exist_ok=True)
 
-    # Download listing files
-    print("Downloading listing files...")
-    run_command(["gh", "release", "download", args.gtiffs_release, "-p", "listing_files.csv", "--clobber"])
-    run_command(["gh", "release", "download", args.pmtiles_release, "-p", args.listing_files_tiled, "--clobber", "-O", "listing_files_tiled.csv"])
-
-    print("Download force redo bounds")
-    force_redo_bounds_file = Path("force_redo_bounds.geojson")
-    try:
-        run_command(["gh", "release", "download", args.pmtiles_release, "-p", f"{force_redo_bounds_file}", "--clobber"])
-        print("Force redo bounds file found and downloaded.")
-    except subprocess.CalledProcessError:
-        print("No force redo bounds file found; proceeding without it.")
-        force_redo_bounds_file = None
-
-    # Determine sheets to retile
-    print("Determining sheets to retile...")
-    with open("listing_files.csv") as f:
-        gtiffs_sheets = {row['name'] for row in csv.DictReader(f)}
-    with open("listing_files_tiled.csv") as f:
-        pmtiles_sheets = {row['name'] for row in csv.DictReader(f)}
-
-    sheets_to_retile = sorted(list(gtiffs_sheets - pmtiles_sheets))
+    # Download bounds files
+    print("Downloading new bounds file from gtiffs release...")
+    run_command(["gh", "release", "download", args.gtiffs_release, "-p", "bounds.geojson", "--clobber", "-O", str(new_bounds_file)])
     
-    with open(retile_list_file, "w") as f:
-        for sheet in sheets_to_retile:
-            f.write(f"{sheet}\n")
-
-    print(f"Found {len(sheets_to_retile)} sheets to retile.")
-    Path("listing_files_tiled.csv").unlink()
+    print("Downloading old bounds file from pmtiles release...")
+    run_command(["gh", "release", "download", args.pmtiles_release, "-p", args.bounds_file_tiled, "--clobber", "-O", str(old_bounds_file)])
+
+    # Create force redo bounds file
+    print("Creating force redo bounds file...")
+    create_force_redo_bounds_main([
+        "--old-bounds-file", str(old_bounds_file),
+        "--new-bounds-file", str(new_bounds_file),
+        "--output-file", str(force_redo_bounds_file)
+    ])
 
-    if not sheets_to_retile:
-        print("No sheets to retile found. Cleaning up and exiting.")
-        Path("listing_files.csv").unlink()
-        retile_list_file.unlink()
+    # Check if there are any changes
+    with open(force_redo_bounds_file) as f:
+        force_redo_data = json.load(f)
+    
+    if not force_redo_data.get("features"):
+        print("No changes detected between bounds files. Exiting.")
         sys.exit(0)
 
-    # Download bounds and original PMTiles
-    print("Downloading bounds file...")
-    run_command(["gh", "release", "download", args.gtiffs_release, "-p", "bounds.geojson"])
+    # Download original PMTiles
     print("Getting original PMTiles files...")
     run_command(["gh", "release", "download", args.pmtiles_release, "-D", str(from_pmtiles_dir), "-p", f"{args.pmtiles_prefix}*"])
 
-    extra_bounds_args = []
-    if force_redo_bounds_file:
-        extra_bounds_args = ["--force-redo-bounds-file", str(force_redo_bounds_file)]
     # Get list of sheets to pull
     print("Getting list of sheets to pull...")
     retile_main([
-        "--retile-list-file", str(retile_list_file),
-        "--bounds-file", "bounds.geojson",
+        "--bounds-file", str(new_bounds_file),
+        "--force-redo-bounds-file", str(force_redo_bounds_file),
         "--sheets-to-pull-list-outfile", str(sheets_to_pull_list_outfile),
         "--from-source", f"{from_pmtiles_prefix}*.pmtiles",
-    ] + extra_bounds_args)
+    ])
 
     print("Sheets to pull:")
     print(sheets_to_pull_list_outfile.read_text())
 
     # Download TIFFs
     print("Downloading TIFFs...")
-    with open("listing_files.csv") as f:
-        url_map = {row[0]: row[2] for row in csv.reader(f)}
-
     with open(sheets_to_pull_list_outfile) as f:
         for line in f:
             fname = line.strip()
-            if fname in url_map:
-                url = url_map[fname]
+            if fname:
                 print(f"Pulling {fname}")
-                run_command(["wget", "-q", "-P", str(tiffs_dir), url])
+                try:
+                    run_command(["gh", "release", "download", args.gtiffs_release, "-p", fname, "-D", str(tiffs_dir), "--clobber"])
+                except subprocess.CalledProcessError as e:
+                    print(f"Failed to download {fname}. It might be a dummy sheet from force redo. Continuing...")
 
     sheets_to_pull_list_outfile.unlink()
 
     # Retile sheets
     print("Retiling the sheets...")
     retile_main([
-        "--retile-list-file", str(retile_list_file),
-        "--bounds-file", "bounds.geojson",
+        "--bounds-file", str(new_bounds_file),
+        "--force-redo-bounds-file", str(force_redo_bounds_file),
         "--from-source", f"{from_pmtiles_prefix}*.pmtiles",
         "--tiles-dir", str(tiles_dir),
         "--tiffs-dir", str(tiffs_dir),
-    ] + extra_bounds_args)
+    ])
 
-    retile_list_file.unlink()
-    Path("bounds.geojson").unlink()
+    old_bounds_file.unlink()
+    force_redo_bounds_file.unlink()
 
     # Create new PMTiles files
     print("Creating new pmtiles files...")
@@ -159,6 +143,7 @@ def cli():
         "--from-source", str(tiles_dir),
         "--from-source", f"{from_pmtiles_prefix}*.pmtiles",
         "--to-pmtiles", f"{to_pmtiles_prefix}.pmtiles",
+        "--exclude-transparent"
     ])
 
     # Upload new PMTiles and manage release assets
@@ -171,20 +156,14 @@ def cli():
     new_pmtiles = glob.glob(f"{to_pmtiles_prefix}*")
     run_command(["gh", "release", "upload", args.pmtiles_release, "--clobber"] + new_pmtiles)
 
-    # Handle listing files
-    if args.listing_files_tiled != "listing_files.csv":
-        print(f"Renaming listing_files.csv to {args.listing_files_tiled}")
-        Path("listing_files.csv").rename(args.listing_files_tiled)
+    # Handle bounds file
+    print("Uploading new bounds file to pmtiles release...")
+    upload_bounds_file = Path(args.bounds_file_tiled)
+    new_bounds_file.rename(upload_bounds_file)
+    
+    run_command(["gh", "release", "upload", args.pmtiles_release, str(upload_bounds_file), "--clobber"])
     
-    print("Uploading new listing file...")
-    run_command(["gh", "release", "upload", args.pmtiles_release, args.listing_files_tiled, "--clobber"])
-    Path(args.listing_files_tiled).unlink()
-
-    # Handle force redo bounds file
-    if force_redo_bounds_file:
-        print("Deleting force redo bounds file...")
-        run_command(["gh", "release", "delete-asset", args.gtiffs_release, str(force_redo_bounds_file), '-y'])
-        force_redo_bounds_file.unlink()
+    upload_bounds_file.unlink()
 
     # Cleanup
     print("Cleaning up staging directories...")