water-column-sonar-processing 25.3.1__tar.gz → 25.8.0__tar.gz

water_column_sonar_processing-25.3.1/LICENSE → water_column_sonar_processing-25.8.0/LICENSE-MIT

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2024 Cooperative Institutes, Coastal and Marine Geophysics
+Copyright (c) 2025 Cooperative Institutes, Coastal and Marine Geophysics
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

water_column_sonar_processing-25.8.0/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: water-column-sonar-processing
+Version: 25.8.0
+Summary: Processing tool for water column sonar data.
+Author-email: Rudy Klucik <rudy.klucik@noaa.gov>
+Maintainer-email: Rudy Klucik <rudy.klucik@noaa.gov>
+License-Expression: MIT
+Project-URL: Homepage, https://echo.fish
+Project-URL: Repository, https://github.com/CI-CMG/water-column-sonar-processing
+Project-URL: Issues, https://github.com/CI-CMG/water-column-sonar-processing/issues
+Keywords: ocean,sonar,water column,zarr
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE-MIT
+Requires-Dist: aiobotocore==2.23.0
+Requires-Dist: boto3==1.38.23
+Requires-Dist: botocore==1.38.23
+Requires-Dist: echopype==0.10.1
+Requires-Dist: fiona==1.10.1
+Requires-Dist: geopandas==1.1.1
+Requires-Dist: mock==5.2.0
+Requires-Dist: moto[all]==5.1.8
+Requires-Dist: moto[server]==5.1.8
+Requires-Dist: multidict==6.6.3
+Requires-Dist: netcdf4==1.6.5
+Requires-Dist: networkx==3.5.0
+Requires-Dist: numcodecs==0.15.1
+Requires-Dist: numpy==1.26.4
+Requires-Dist: pandas==2.3.0
+Requires-Dist: pooch==1.8.2
+Requires-Dist: pyarrow==20.0.0
+Requires-Dist: pykalman==0.10.1
+Requires-Dist: python-dotenv==1.1.1
+Requires-Dist: requests==2.32.4
+Requires-Dist: s3fs==2025.5.1
+Requires-Dist: scipy==1.15.1
+Requires-Dist: setuptools==80.9.0
+Requires-Dist: shapely==2.1.1
+Requires-Dist: xarray==2025.7.0
+Requires-Dist: xbatcher[tensorflow]==0.4.0
+Requires-Dist: zarr==2.18.7
+Provides-Extra: dev
+Requires-Dist: bandit[toml]==1.8.0; extra == "dev"
+Requires-Dist: bandit; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: isort; extra == "dev"
+Requires-Dist: matplotlib; extra == "dev"
+Requires-Dist: pooch==1.8.2; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: pyinstaller; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: tqdm; extra == "dev"
+Dynamic: license-file
+
+# Water Column Sonar Processing
+Processing tool for converting Level_0 water column sonar data to Level_1 and Level_2 derived data sets as well as generating geospatial information.
+
+![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/CI-CMG/water-column-sonar-processing/test_action.yaml)
+![PyPI - Implementation](https://img.shields.io/pypi/v/ci-cmg-water-column-sonar-processing) ![GitHub License](https://img.shields.io/github/license/CI-CMG/water-column-sonar-processing) ![PyPI - Downloads](https://img.shields.io/pypi/dd/ci-cmg-water-column-sonar-processing) ![GitHub code size in bytes](https://img.shields.io/github/languages/code-size/CI-CMG/water-column-sonar-processing) ![GitHub repo size](https://img.shields.io/github/repo-size/CI-CMG/water-column-sonar-processing)
+
+# Setting up the Python Environment
+> Python 3.12.11
+
+# Installing Dependencies
+```
+source .venv/bin/activate
+
+uv pip install --upgrade pip
+
+uv pip install -r pyproject.toml --all-extras
+
+uv run pre-commit install
+```
+
+# Pytest
+```
+uv run pytest --cache-clear tests -W ignore::DeprecationWarning
+```
+or
+> pytest --cache-clear --cov=src tests/ --cov-report=xml
+
+# Instructions
+Following this tutorial:
+https://packaging.python.org/en/latest/tutorials/packaging-projects/
+
+# Pre Commit Hook
+see here for installation: https://pre-commit.com/
+https://dev.to/rafaelherik/using-trufflehog-and-pre-commit-hook-to-prevent-secret-exposure-edo
+```
+uv run pre-commit install --allow-missing-config
+# or
+uv run pre-commit install
+```
+
+# Black
+https://ljvmiranda921.github.io/notebook/2018/06/21/precommits-using-black-and-flake8/
+```
+Settings > Black
+Execution mode: Package
+Python Interpreter: .../.venv/bin/python
+Use Black Formatter: X On Code reformat, X On Save
+```
+
+# Linting
+Ruff
+https://plugins.jetbrains.com/plugin/20574-ruff
+
+# Colab Test
+https://colab.research.google.com/drive/1KiLMueXiz9WVB9o4RuzYeGjNZ6PsZU7a#scrollTo=AayVyvpBdfIZ
+
+# Test Coverage
+TODO
+
+# Tag a Release
+Step 1 --> increment the semantic version in the zarr_manager.py "metadata" & the "pyproject.toml"
+```commandline
+git tag -a v25.8.0 -m "Releasing v25.8.0"
+git push origin --tags
+```
+
+# To Publish To PROD
+```
+uv build --no-sources
+uv publish
+```
+
+# TODO:
+add https://pypi.org/project/setuptools-scm/
+for extracting the version
+
+# Security scanning
+> bandit -r water_column_sonar_processing/
+
+# Data Debugging
+Experimental Plotting in Xarray (hvPlot):
+https://colab.research.google.com/drive/18vrI9LAip4xRGEX6EvnuVFp35RAiVYwU#scrollTo=q9_j9p2yXsLV
+
+HB0707 Zoomable Cruise:
+https://hb0707.s3.us-east-1.amazonaws.com/index.html
+
+
+# UV Debugging
+```
+uv lock --check
+uv lock
+uv sync --extra dev
+```
+
+# Fixing S3FS Problems
+```commandline
+To enable/disa asyncio for the debugger, follow the steps:
+Open PyCharm
+Use Shift + Shift (Search Everywhere)
+In the popup type: Registry and press Enter
+Find "Registry" in the list of results and click on it.
+In the new popup find python.debug.asyncio.repl line and check the respective checkbox
+Press Close.
+Restart the IDE.
+The asyncio support will be enabled in the debugger.
+```

water_column_sonar_processing-25.8.0/README.md

@@ -0,0 +1,106 @@
+# Water Column Sonar Processing
+Processing tool for converting Level_0 water column sonar data to Level_1 and Level_2 derived data sets as well as generating geospatial information.
+
+![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/CI-CMG/water-column-sonar-processing/test_action.yaml)
+![PyPI - Implementation](https://img.shields.io/pypi/v/ci-cmg-water-column-sonar-processing) ![GitHub License](https://img.shields.io/github/license/CI-CMG/water-column-sonar-processing) ![PyPI - Downloads](https://img.shields.io/pypi/dd/ci-cmg-water-column-sonar-processing) ![GitHub code size in bytes](https://img.shields.io/github/languages/code-size/CI-CMG/water-column-sonar-processing) ![GitHub repo size](https://img.shields.io/github/repo-size/CI-CMG/water-column-sonar-processing)
+
+# Setting up the Python Environment
+> Python 3.12.11
+
+# Installing Dependencies
+```
+source .venv/bin/activate
+
+uv pip install --upgrade pip
+
+uv pip install -r pyproject.toml --all-extras
+
+uv run pre-commit install
+```
+
+# Pytest
+```
+uv run pytest --cache-clear tests -W ignore::DeprecationWarning
+```
+or
+> pytest --cache-clear --cov=src tests/ --cov-report=xml
+
+# Instructions
+Following this tutorial:
+https://packaging.python.org/en/latest/tutorials/packaging-projects/
+
+# Pre Commit Hook
+see here for installation: https://pre-commit.com/
+https://dev.to/rafaelherik/using-trufflehog-and-pre-commit-hook-to-prevent-secret-exposure-edo
+```
+uv run pre-commit install --allow-missing-config
+# or
+uv run pre-commit install
+```
+
+# Black
+https://ljvmiranda921.github.io/notebook/2018/06/21/precommits-using-black-and-flake8/
+```
+Settings > Black
+Execution mode: Package
+Python Interpreter: .../.venv/bin/python
+Use Black Formatter: X On Code reformat, X On Save
+```
+
+# Linting
+Ruff
+https://plugins.jetbrains.com/plugin/20574-ruff
+
+# Colab Test
+https://colab.research.google.com/drive/1KiLMueXiz9WVB9o4RuzYeGjNZ6PsZU7a#scrollTo=AayVyvpBdfIZ
+
+# Test Coverage
+TODO
+
+# Tag a Release
+Step 1 --> increment the semantic version in the zarr_manager.py "metadata" & the "pyproject.toml"
+```commandline
+git tag -a v25.8.0 -m "Releasing v25.8.0"
+git push origin --tags
+```
+
+# To Publish To PROD
+```
+uv build --no-sources
+uv publish
+```
+
+# TODO:
+add https://pypi.org/project/setuptools-scm/
+for extracting the version
+
+# Security scanning
+> bandit -r water_column_sonar_processing/
+
+# Data Debugging
+Experimental Plotting in Xarray (hvPlot):
+https://colab.research.google.com/drive/18vrI9LAip4xRGEX6EvnuVFp35RAiVYwU#scrollTo=q9_j9p2yXsLV
+
+HB0707 Zoomable Cruise:
+https://hb0707.s3.us-east-1.amazonaws.com/index.html
+
+
+# UV Debugging
+```
+uv lock --check
+uv lock
+uv sync --extra dev
+```
+
+# Fixing S3FS Problems
+```commandline
+To enable/disa asyncio for the debugger, follow the steps:
+Open PyCharm
+Use Shift + Shift (Search Everywhere)
+In the popup type: Registry and press Enter
+Find "Registry" in the list of results and click on it.
+In the new popup find python.debug.asyncio.repl line and check the respective checkbox
+Press Close.
+Restart the IDE.
+The asyncio support will be enabled in the debugger.
+```

water_column_sonar_processing-25.8.0/pyproject.toml

@@ -0,0 +1,90 @@
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "water-column-sonar-processing"
+version = "25.8.0"
+authors = [
+  { name="Rudy Klucik", email="rudy.klucik@noaa.gov" },
+]
+maintainers = [
+  { name="Rudy Klucik", email="rudy.klucik@noaa.gov" },
+]
+description = "Processing tool for water column sonar data."
+keywords = ["ocean", "sonar", "water column", "zarr"]
+readme = "README.md"
+requires-python = ">=3.12"
+license = "MIT"
+license-files = ["LICENSE-MIT"]
+
+dependencies = [
+    "aiobotocore==2.23.0",
+#    "aiohttp[speedups]",
+    "boto3==1.38.23",
+    "botocore==1.38.23",
+    "echopype==0.10.1",
+    "fiona==1.10.1",
+    "geopandas==1.1.1",
+#    "keras==3.9.2",
+    "mock==5.2.0",
+    "moto[all]==5.1.8",
+    "moto[server]==5.1.8",
+    "multidict==6.6.3",
+    "netcdf4==1.6.5",
+    "networkx==3.5.0",
+    "numcodecs==0.15.1", # problem with blosc
+    "numpy==1.26.4",
+    "pandas==2.3.0",
+    #"pelicanfs==1.0.2", # TODO: add this
+    "pooch==1.8.2",
+    "pyarrow==20.0.0",
+    "pykalman==0.10.1",
+    "python-dotenv==1.1.1",
+    "requests==2.32.4",
+    "s3fs==2025.5.1",
+    "scipy==1.15.1",
+    "setuptools==80.9.0",
+    "shapely==2.1.1",
+#    "tensorflow==2.18.0",
+    "xarray==2025.7.0",
+    "xbatcher[tensorflow]==0.4.0",
+    "zarr==2.18.7",
+]
+
+[project.optional-dependencies]
+dev = [
+    "bandit[toml]==1.8.0",
+    "bandit",
+    "black",
+    "build",
+    "isort",
+    "matplotlib",
+    "pooch==1.8.2",
+    "pre-commit",
+    "pyinstaller",
+    "pytest",
+    "ruff",
+    "tqdm",
+]
+
+[project.urls]
+Homepage = "https://echo.fish"
+Repository = "https://github.com/CI-CMG/water-column-sonar-processing"
+Issues = "https://github.com/CI-CMG/water-column-sonar-processing/issues"
+
+[tool.bandit]
+exclude_dirs = ["tests"]
+
+[tool.pre-commit-hooks.bandit]
+exclude = ["*/tests/*"]
+
+[tool.isort]
+known_first_party = "water_column_sonar_processing"
+multi_line_output = 3
+force_grid_wrap = 0
+include_trailing_comma = true
+combine_as_imports = true
+
+[tool.black]
+line-length = 88

{water_column_sonar_processing-25.3.1 → water_column_sonar_processing-25.8.0}/water_column_sonar_processing/aws/dynamodb_manager.py

@@ -127,7 +127,7 @@ class DynamoDBManager:
             # assert status_code == 200, "Problem, unable to update dynamodb table."
             # assert response['ConsumedCapacity']['TableName'] == table_name
         except Exception as err:
-            print(f"Problem was encountered while updating item: {err}")
+            raise RuntimeError(f"Problem was encountered while updating item, {err}")
 
     #####################################################################
     # TODO: change to "get_cruise_as_df"
@@ -135,7 +135,7 @@ class DynamoDBManager:
         self,
         # ship_name,
         cruise_name,
-        # sensor_name,
+        # sensor_name, # TODO: need to add this back for EK80
         table_name,
     ) -> pd.DataFrame:
         """
@@ -230,7 +230,7 @@ class DynamoDBManager:
     #     if len(response["Items"]) == 0 and "LastEvaluatedKey" not in response:
     #         return pd.DataFrame() # If no results, return empty dataframe
     #
-    #     data = response["Items"]
+    #     dataset = response["Items"]
     #
     #     while response.get('LastEvaluatedKey'): #"LastEvaluatedKey" in response:
     #         response = self.dynamodb_client.scan(
@@ -252,10 +252,10 @@ class DynamoDBManager:
     #             ConsistentRead=True,
     #             ExclusiveStartKey=response["LastEvaluatedKey"],
     #         )
-    #         data.extend(response["Items"])
+    #         dataset.extend(response["Items"])
     #
     #     deserializer = self.type_deserializer
-    #     df = pd.DataFrame([deserializer.deserialize({"M": i}) for i in data])
+    #     df = pd.DataFrame([deserializer.deserialize({"M": i}) for i in dataset])
     #
     #     return df.sort_values(by="START_TIME", ignore_index=True)
 
@@ -273,7 +273,7 @@ class DynamoDBManager:
         response = self.dynamodb_client.delete_item(
             Key={"CRUISE_NAME": {"S": cruise_name}, "FILE_NAME": {"S": file_name}},
             TableName=table_name,
-            ReturnConsumedCapacity="TOTALS",
+            ReturnConsumedCapacity="TOTAL",
         )
         # TODO: there should be attributes included in response but they are missing
         # if response["ResponseMetadata"]["HTTPStatusCode"] != 200: