asp-plot 0.0.1__tar.gz

asp_plot-0.0.1/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2022, UW Terrain Analysis and Cryosphere Observation Lab
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

asp_plot-0.0.1/PKG-INFO

@@ -0,0 +1,76 @@
+Metadata-Version: 2.1
+Name: asp_plot
+Version: 0.0.1
+Summary: Package for plotting outputs Ames Stereo Pipeline processing
+Author: Ben Purinton
+Author-email: Ben Purinton <purinton@uw.edu>
+Project-URL: Homepage, https://github.com/uw-cryo/asp_plot
+Project-URL: Issues, https://github.com/uw-cryo/asp_plot/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+# asp_plot
+
+Scripts and notebooks to visualize output from the [NASA Ames Stereo Pipeline (ASP)](https://github.com/NeoGeographyToolkit/StereoPipeline).
+
+## Motivation
+
+Our objective is to release a modular Python package with a command-line interface (CLI) that can be run automatically on an ASP output directory to prepare a set of standard diagnostic plots, publication-quality output figures, and a pdf report with relevant information, similar to the reports prepared by many commercial SfM software packages (e.g., Agisoft Metashape, Pix4DMapper).
+
+
+## Status
+
+This is a work in progress.
+
+The directory `original_code/` contains initial notebooks compiled from recent projects using sample stereo images from the Maxar WorldView, Planet SkySat-C and BlackSky Global constellations. 
+
+The functionality of these notebooks is being ported to the `asp_plot/` directory, which is the package `asp_plot`.
+
+
+## Installing and testing the package
+
+```
+$ git clone git@github.com:uw-cryo/asp_plot.git
+$ cd asp_plot
+$ conda env create -f environment.yml
+$ conda activate asp_plot
+$ pip install -e .
+$ python3 setup.py install
+```
+
+To ensure the install was successful, tests can be run with:
+
+```
+$ pytest
+```
+
+## Notebook example usage
+
+Examples of the modular usage of the package can be found in the `notebooks/` directory.
+
+
+## CLI usage
+
+A full report and individual plots can be output via the command-line:
+
+```
+$ asp_plot --directory ./asp_processing \
+           --bundle_adjust_directory ba \
+           --stereo_directory stereo \
+           --map_crs EPSG:32604 \
+           --reference_dem ref_dem.tif \
+           --plots_directory asp_plots \
+           --report_filename asp_plot_report.pdf
+```
+
+Use:
+
+```
+$ asp_plot --help
+```
+
+for details (and defaults) of the command-line flags.

asp_plot-0.0.1/README.md

@@ -0,0 +1,61 @@
+# asp_plot
+
+Scripts and notebooks to visualize output from the [NASA Ames Stereo Pipeline (ASP)](https://github.com/NeoGeographyToolkit/StereoPipeline).
+
+## Motivation
+
+Our objective is to release a modular Python package with a command-line interface (CLI) that can be run automatically on an ASP output directory to prepare a set of standard diagnostic plots, publication-quality output figures, and a pdf report with relevant information, similar to the reports prepared by many commercial SfM software packages (e.g., Agisoft Metashape, Pix4DMapper).
+
+
+## Status
+
+This is a work in progress.
+
+The directory `original_code/` contains initial notebooks compiled from recent projects using sample stereo images from the Maxar WorldView, Planet SkySat-C and BlackSky Global constellations. 
+
+The functionality of these notebooks is being ported to the `asp_plot/` directory, which is the package `asp_plot`.
+
+
+## Installing and testing the package
+
+```
+$ git clone git@github.com:uw-cryo/asp_plot.git
+$ cd asp_plot
+$ conda env create -f environment.yml
+$ conda activate asp_plot
+$ pip install -e .
+$ python3 setup.py install
+```
+
+To ensure the install was successful, tests can be run with:
+
+```
+$ pytest
+```
+
+## Notebook example usage
+
+Examples of the modular usage of the package can be found in the `notebooks/` directory.
+
+
+## CLI usage
+
+A full report and individual plots can be output via the command-line:
+
+```
+$ asp_plot --directory ./asp_processing \
+           --bundle_adjust_directory ba \
+           --stereo_directory stereo \
+           --map_crs EPSG:32604 \
+           --reference_dem ref_dem.tif \
+           --plots_directory asp_plots \
+           --report_filename asp_plot_report.pdf
+```
+
+Use:
+
+```
+$ asp_plot --help
+```
+
+for details (and defaults) of the command-line flags.

asp_plot-0.0.1/asp_plot/__init__.py

@@ -0,0 +1,5 @@
+import asp_plot.utils
+import asp_plot.processing_parameters
+import asp_plot.scenes
+import asp_plot.bundle_adjust
+import asp_plot.stereo

asp_plot-0.0.1/asp_plot/bundle_adjust.py

@@ -0,0 +1,212 @@
+import os
+import glob
+import pandas as pd
+import geopandas as gpd
+import matplotlib.pyplot as plt
+from matplotlib.lines import Line2D
+import contextily as ctx
+from asp_plot.utils import ColorBar, Plotter, save_figure
+
+
+class ReadResiduals:
+    def __init__(self, directory, bundle_adjust_directory):
+        self.directory = directory
+        self.bundle_adjust_directory = bundle_adjust_directory
+
+    def get_init_final_residuals_csvs(self):
+        filenames = [
+            "*-initial_residuals_pointmap.csv",
+            "*-final_residuals_pointmap.csv",
+        ]
+
+        paths = [
+            glob.glob(
+                os.path.join(self.directory, self.bundle_adjust_directory, filename)
+            )[0]
+            for filename in filenames
+        ]
+
+        for path in paths:
+            if not os.path.isfile(path):
+                raise ValueError(f"Residuals CSV file not found: {path}")
+
+        resid_init_path, resid_final_path = paths
+        return resid_init_path, resid_final_path
+
+    def get_residuals_gdf(self, csv_path):
+        cols = [
+            "lon",
+            "lat",
+            "height_above_datum",
+            "mean_residual",
+            "num_observations",
+        ]
+
+        resid_df = pd.read_csv(csv_path, skiprows=2, names=cols)
+
+        # Need the astype('str') to handle cases where column has dtype of int (without the # from DEM appended to some rows)
+        resid_df["from_DEM"] = (
+            resid_df["num_observations"].astype("str").str.contains("# from DEM")
+        )
+
+        resid_df["num_observations"] = (
+            resid_df["num_observations"]
+            .astype("str")
+            .str.split("#", expand=True)[0]
+            .astype(int)
+        )
+
+        resid_gdf = gpd.GeoDataFrame(
+            resid_df,
+            geometry=gpd.points_from_xy(
+                resid_df["lon"], resid_df["lat"], crs="EPSG:4326"
+            ),
+        )
+
+        resid_gdf.filename = os.path.basename(csv_path)
+        return resid_gdf
+
+    def get_init_final_residuals_gdfs(self):
+        resid_init_path, resid_final_path = self.get_init_final_residuals_csvs()
+        resid_init_gdf = self.get_residuals_gdf(resid_init_path)
+        resid_final_gdf = self.get_residuals_gdf(resid_final_path)
+        return resid_init_gdf, resid_final_gdf
+
+    def get_mapproj_residuals_gdf(self):
+        path = glob.glob(
+            os.path.join(
+                self.directory,
+                self.bundle_adjust_directory,
+                "*-mapproj_match_offsets.txt",
+            )
+        )[0]
+        if not os.path.isfile(path):
+            raise ValueError(f"MapProj Residuals TXT file not found: {path}")
+
+        cols = ["lon", "lat", "height_above_datum", "mapproj_ip_dist_meters"]
+        resid_mapprojected_df = pd.read_csv(path, skiprows=2, names=cols)
+        resid_mapprojected_gdf = gpd.GeoDataFrame(
+            resid_mapprojected_df,
+            geometry=gpd.points_from_xy(
+                resid_mapprojected_df["lon"],
+                resid_mapprojected_df["lat"],
+                crs="EPSG:4326",
+            ),
+        )
+        return resid_mapprojected_gdf
+
+    def get_propagated_triangulation_uncert_df(self):
+        path = glob.glob(
+            os.path.join(
+                self.directory,
+                self.bundle_adjust_directory,
+                "*-triangulation_uncertainty.txt",
+            )
+        )[0]
+        if not os.path.isfile(path):
+            raise ValueError(f"Triangulation Uncertainty TXT file not found: {path}")
+
+        cols = [
+            "left_image",
+            "right_image",
+            "horiz_error_median",
+            "vert_error_median",
+            "horiz_error_mean",
+            "vert_error_mean",
+            "horiz_error_stddev",
+            "vert_error_stddev",
+            "num_meas",
+        ]
+        resid_triangulation_uncert_df = pd.read_csv(
+            path, sep=" ", skiprows=2, names=cols
+        )
+        return resid_triangulation_uncert_df
+
+
+class PlotResiduals(Plotter):
+    def __init__(self, geodataframes, **kwargs):
+        super().__init__(**kwargs)
+        if not isinstance(geodataframes, list):
+            raise ValueError("Input must be a list of GeoDataFrames")
+        self.geodataframes = geodataframes
+
+    def get_residual_stats(self, gdf, column_name="mean_residual"):
+        stats = gdf[column_name].quantile([0.25, 0.50, 0.84, 0.95]).round(2).tolist()
+        return stats
+
+    def plot_n_residuals(
+        self,
+        column_name="mean_residual",
+        cbar_label="Mean Residual (m)",
+        clip_final=True,
+        clim=None,
+        common_clim=True,
+        cmap="inferno",
+        map_crs="EPSG:4326",
+        save_dir=None,
+        fig_fn=None,
+        **ctx_kwargs,
+    ):
+
+        # Get rows and columns and create subplots
+        n = len(self.geodataframes)
+        nrows = (n + 3) // 4
+        ncols = min(n, 4)
+        if n == 1:
+            fig, axa = plt.subplots(1, 1, figsize=(8, 6))
+            axa = [axa]
+        else:
+            fig, axa = plt.subplots(
+                nrows, ncols, figsize=(4 * ncols, 3 * nrows), sharex=True, sharey=True
+            )
+            axa = axa.flatten()
+
+        # Plot each GeoDataFrame
+        for i, gdf in enumerate(self.geodataframes):
+            gdf = gdf.sort_values(by=column_name).to_crs(map_crs)
+
+            if clim is None:
+                clim = ColorBar().get_clim(gdf[column_name])
+
+            if common_clim:
+                self.plot_geodataframe(
+                    ax=axa[i],
+                    gdf=gdf,
+                    clim=clim,
+                    column_name=column_name,
+                    cbar_label=cbar_label,
+                )
+            else:
+                self.plot_geodataframe(
+                    ax=axa[i], gdf=gdf, column_name=column_name, cbar_label=cbar_label
+                )
+
+            ctx.add_basemap(ax=axa[i], **ctx_kwargs)
+
+            if clip_final and i == n - 1:
+                axa[i].autoscale(False)
+
+            # Show some statistics and information
+            stats = self.get_residual_stats(gdf, column_name)
+            stats_text = f"(n={gdf.shape[0]})\n" + "\n".join(
+                f"{quantile*100:.0f}th: {stat}"
+                for quantile, stat in zip([0.25, 0.50, 0.84, 0.95], stats)
+            )
+            axa[i].text(
+                0.05,
+                0.95,
+                stats_text,
+                transform=axa[i].transAxes,
+                fontsize=8,
+                verticalalignment="top",
+                bbox=dict(boxstyle="round", facecolor="white", alpha=0.8),
+            )
+
+        # Clean up axes and tighten layout
+        for i in range(n, nrows * ncols):
+            fig.delaxes(axa[i])
+        fig.suptitle(self.title, size=10)
+        plt.subplots_adjust(wspace=0.2, hspace=0.4)
+        fig.tight_layout()
+        if save_dir and fig_fn:
+            save_figure(fig, save_dir, fig_fn)

asp_plot-0.0.1/asp_plot/cli/asp_plot.py

@@ -0,0 +1,184 @@
+import os
+import glob
+import subprocess
+import click
+import contextily as ctx
+from asp_plot.processing_parameters import ProcessingParameters
+from asp_plot.scenes import ScenePlotter
+from asp_plot.bundle_adjust import ReadResiduals, PlotResiduals
+from asp_plot.stereo import StereoPlotter
+from asp_plot.utils import compile_report
+
+
+@click.command()
+@click.option(
+    "--directory",
+    prompt=True,
+    default="./",
+    help="Directory of ASP processing with scenes and sub-directories for bundle adjustment and stereo. Default: current directory",
+)
+@click.option(
+    "--bundle_adjust_directory",
+    prompt=True,
+    default="ba",
+    help="Directory of bundle adjustment files. Default: ba",
+)
+@click.option(
+    "--stereo_directory",
+    prompt=True,
+    default="stereo",
+    help="Directory of stereo files. Default: stereo",
+)
+@click.option(
+    "--map_crs",
+    prompt=True,
+    default="EPSG:4326",
+    help="Projection for bundle adjustment plots. Default: EPSG:4326",
+)
+@click.option(
+    "--reference_dem",
+    prompt=True,
+    default="",
+    help="Reference DEM used in ASP processing. Default: ",
+)
+@click.option(
+    "--plots_directory",
+    prompt=True,
+    default="asp_plots",
+    help="Directory to put output plots. Default: asp_plots",
+)
+@click.option(
+    "--report_filename",
+    prompt=True,
+    default="asp_plot_report.pdf",
+    help="PDF file to write out for report. Default: asp_plot_report.pdf",
+)
+def main(
+    directory,
+    bundle_adjust_directory,
+    stereo_directory,
+    map_crs,
+    reference_dem,
+    plots_directory,
+    report_filename,
+):
+    print(f"\n\nProcessing ASP files in {directory}\n\n")
+
+    plots_directory = os.path.join(directory, plots_directory)
+    os.makedirs(plots_directory, exist_ok=True)
+    report_pdf_path = os.path.join(directory, report_filename)
+
+    # Geometry plot
+    try:
+        subprocess.run(["dg_geom_plot.py", directory])
+        subprocess.run(
+            f"mv {directory}/*stereo_geom.png {plots_directory}/00_stereo_geom.png",
+            shell=True,
+        )
+    except:
+        print(
+            "Could not generate stereo geometry plot, check your path for dg_geom_plot.py"
+        )
+
+    # Scene plot
+    plotter = ScenePlotter(directory, stereo_directory, title="Mapprojected Scenes")
+
+    plotter.plot_orthos(save_dir=plots_directory, fig_fn="01_orthos.png")
+
+    # Bundle adjustment plots
+    residuals = ReadResiduals(directory, bundle_adjust_directory)
+    resid_init_gdf, resid_final_gdf = residuals.get_init_final_residuals_gdfs()
+    resid_mapprojected_gdf = residuals.get_mapproj_residuals_gdf()
+
+    ctx_kwargs = {
+        "crs": map_crs,
+        "source": ctx.providers.Esri.WorldImagery,
+        "attribution_size": 0,
+        "alpha": 0.5,
+    }
+
+    plotter = PlotResiduals(
+        [resid_init_gdf, resid_final_gdf],
+        lognorm=True,
+        title="Bundle Adjust Initial and Final Residuals (Log Scale)",
+    )
+
+    plotter.plot_n_residuals(
+        column_name="mean_residual",
+        cbar_label="Mean Residual (m)",
+        map_crs=map_crs,
+        save_dir=plots_directory,
+        fig_fn="02_ba_residuals_log.png",
+        **ctx_kwargs,
+    )
+
+    plotter.lognorm = False
+    plotter.title = "Bundle Adjust Initial and Final Residuals (Linear Scale)"
+
+    plotter.plot_n_residuals(
+        column_name="mean_residual",
+        cbar_label="Mean Residual (m)",
+        common_clim=False,
+        map_crs=map_crs,
+        save_dir=plots_directory,
+        fig_fn="03_ba_residuals_linear.png",
+        **ctx_kwargs,
+    )
+
+    plotter = PlotResiduals(
+        [resid_mapprojected_gdf],
+        title="Bundle Adjust Midpoint distance between\nfinal interest points projected onto reference DEM",
+    )
+
+    plotter.plot_n_residuals(
+        column_name="mapproj_ip_dist_meters",
+        cbar_label="Interest point distance (m)",
+        map_crs=map_crs,
+        save_dir=plots_directory,
+        fig_fn="04_ba_residuals_mapproj_dist.png",
+        **ctx_kwargs,
+    )
+
+    # Stereo plots
+    plotter = StereoPlotter(
+        directory,
+        stereo_directory,
+        reference_dem,
+        out_dem_gsd=1,
+        title="Stereo Match Points",
+    )
+
+    plotter.plot_match_points(
+        save_dir=plots_directory,
+        fig_fn="05_stereo_match_points.png",
+    )
+
+    plotter.title = "Disparity (pixels)"
+
+    plotter.plot_disparity(
+        unit="pixels",
+        quiver=True,
+        save_dir=plots_directory,
+        fig_fn="06_disparity_pixels.png",
+    )
+
+    plotter.title = "Stereo DEM Results"
+
+    plotter.plot_dem_results(
+        save_dir=plots_directory,
+        fig_fn="07_stereo_dem_results.png",
+    )
+
+    # Compile report
+    processing_parameters = ProcessingParameters(
+        directory, bundle_adjust_directory, stereo_directory
+    )
+    processing_parameters_dict = processing_parameters.from_log_files()
+
+    compile_report(plots_directory, processing_parameters_dict, report_pdf_path)
+
+    print(f"\n\nReport saved to {report_pdf_path}\n\n")
+
+
+if __name__ == "__main__":
+    main()

asp_plot-0.0.1/asp_plot/processing_parameters.py

@@ -0,0 +1,121 @@
+import os
+import glob
+import matplotlib.pyplot as plt
+from asp_plot.utils import save_figure
+
+
+class ProcessingParameters:
+    def __init__(self, directory, bundle_adjust_directory, stereo_directory):
+        self.directory = directory
+        self.bundle_adjust_directory = bundle_adjust_directory
+        self.stereo_directory = stereo_directory
+        self.processing_parameters_dict = {}
+
+        try:
+            self.bundle_adjust_log = glob.glob(
+                os.path.join(self.directory, self.bundle_adjust_directory, "*log*.txt")
+            )[0]
+            self.stereo_log = glob.glob(
+                os.path.join(self.directory, self.stereo_directory, "*log-stereo*.txt")
+            )[0]
+            self.point2dem_log = glob.glob(
+                os.path.join(
+                    self.directory, self.stereo_directory, "*log-point2dem*.txt"
+                )
+            )[0]
+        except:
+            raise ValueError(
+                "Could not find log files in bundle adjust and stereo directories\nCheck that these *log*.txt files exist in the directories specified"
+            )
+
+    def from_log_files(self):
+        with open(self.bundle_adjust_log, "r") as file:
+            content = file.readlines()
+
+        bundle_adjust_params = ""
+        processing_timestamp = ""
+
+        for line in content:
+            if "bundle_adjust" in line and not bundle_adjust_params:
+                bundle_adjust_params = line.strip()
+
+            if "[ console ]" in line and not processing_timestamp:
+                date, time = line.split()[0], line.split()[1]
+                processing_timestamp = f"{date}-{time[:5].replace(':', '')}"
+
+            if bundle_adjust_params and processing_timestamp:
+                break
+
+        with open(self.stereo_log, "r") as file:
+            content = file.readlines()
+
+        stereo_params = ""
+
+        for line in content:
+            if "stereo" in line and not stereo_params:
+                stereo_params = line.strip()
+
+            if stereo_params:
+                break
+
+        with open(self.point2dem_log, "r") as file:
+            content = file.readlines()
+
+        point2dem_params = ""
+
+        for line in content:
+            if "point2dem" in line and not point2dem_params:
+                point2dem_params = line.strip()
+
+            if point2dem_params:
+                break
+
+        self.processing_parameters_dict = {
+            "bundle_adjust": bundle_adjust_params,
+            "stereo": stereo_params,
+            "point2dem": point2dem_params,
+            "processing_timestamp": processing_timestamp,
+        }
+
+        return self.processing_parameters_dict
+
+    def plot_processing_parameters(self, save_dir=None, fig_fn=None):
+        fig, axes = plt.subplots(3, 1, figsize=(10, 6))
+        ax1, ax2, ax3 = axes.flatten()
+
+        ax1.axis("off")
+        ax1.text(
+            0.5,
+            0.5,
+            f"Processed on: {self.processing_parameters_dict['processing_timestamp']:}\n\nBundle Adjust:\n{self.processing_parameters_dict['bundle_adjust']:}",
+            horizontalalignment="center",
+            verticalalignment="center",
+            fontsize=10,
+            wrap=True,
+        )
+
+        ax2.axis("off")
+        ax2.text(
+            0.5,
+            0.5,
+            f"Stereo:\n{self.processing_parameters_dict['stereo']:}",
+            horizontalalignment="center",
+            verticalalignment="center",
+            fontsize=10,
+            wrap=True,
+        )
+
+        ax3.axis("off")
+        ax3.text(
+            0.5,
+            0.5,
+            f"point2dem:\n{self.processing_parameters_dict['point2dem']:}",
+            horizontalalignment="center",
+            verticalalignment="center",
+            fontsize=10,
+            wrap=True,
+        )
+
+        fig.tight_layout()
+        if save_dir and fig_fn:
+            save_figure(fig, save_dir, fig_fn)