water-column-sonar-processing 0.0.1__tar.gz

water_column_sonar_processing-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Cooperative Institutes, Coastal and Marine Geophysics
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

water_column_sonar_processing-0.0.1/PKG-INFO

@@ -0,0 +1,89 @@
+Metadata-Version: 2.1
+Name: water-column-sonar-processing
+Version: 0.0.1
+Summary: A processing tool for water column sonar data.
+Author-email: Rudy Klucik <rudy.klucik@noaa.gov>
+Project-URL: Homepage, https://github.com/CI-CMG/water-column-sonar-processing
+Project-URL: Issues, https://github.com/CI-CMG/water-column-sonar-processing/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+# water-column-sonar-processing
+Processing tool for converting L0 data to L1 and L2 as well as generating geospatial information
+
+## Setting up the Python Environment
+> Python 3.10.12
+
+# MacOS Pyenv Installation Instructions
+  1. Install pyenv (https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv)
+     1. ```brew update```
+     2. ```arch -arm64 brew install pyenv```
+     3. In ~/.bashrc add
+        1. ```export PYENV_ROOT="$HOME/.pyenv"```
+        2. ```export PATH="$PYENV_ROOT/bin:$PATH"```
+        3. ```eval "$(pyenv init -)"```
+     4. ```arch -arm64 brew install openssl readline sqlite3 xz zlib tcl-tk```
+  2. Install pyenv-virtualenv (https://github.com/pyenv/pyenv-virtualenv)
+     1. ```arch -arm64 brew install pyenv-virtualenv```
+     2. In ~/.bashrc add
+         1. ```eval "$(pyenv virtualenv-init -)"```
+  3. Open a new terminal
+  4. Install Python version
+     1. ```env PYTHON_CONFIGURE_OPTS="--enable-shared"```
+     2. ```env CONFIGURE_OPTS='--enable-optimizations' arch -arm64 pyenv install 3.10.12```
+  5. Create virtual env (to delete 'pyenv uninstall 3.10.12/water-column-sonar-processing')
+     1. ```pyenv virtualenv 3.10.12 water-column-sonar-processing```
+  6. Set local version of python (if not done already)
+     1. change directory to root of project
+     2. ```pyenv local 3.10.12 water-column-sonar-processing```
+     3. ```pyenv activate water-column-sonar-processing```
+
+## Setting up IntelliJ
+
+  1. Install the IntelliJ Python plugin
+  2. Set up pyenv
+     1. File -> Project Structure or CMD + ;
+     2. SDKs -> + -> Add Python SDK -> Virtual Environment
+     3. Select Existing Environment
+     4. Choose ~/.pyenv/versions/mocking_aws/bin/python
+  3. Set up Python Facet (not sure if this is required)
+     1. File -> Project Structure or CMD + ;
+     2. Facets -> + -> Python 
+     3. Set interpreter 
+
+## Installing Dependencies
+
+  1. Add dependencies with versions to requirements.txt
+  2. ```pip install --upgrade pip && pip install -r requirements_dev.txt```
+
+
+## Pytest
+```commandline
+pytest --disable-warnings
+```
+
+## Instructions
+Following this tutorial:
+https://packaging.python.org/en/latest/tutorials/packaging-projects/
+
+## To Publish To TEST
+```commandline
+python -m build
+python -m twine upload --repository testpypi dist/*
+pytho -m pip install --index-url https://test.pypi.org/simple/ hello-pypi-rudy-klucik
+python
+```
+```python
+from water-column-sonar-processing import ZarrManager
+example.add_one(2)
+```
+
+## To Publish To PROD
+```commandline
+python -m build
+python -m twine upload --repository pypi dist/*
+```

water_column_sonar_processing-0.0.1/README.md

@@ -0,0 +1,75 @@
+# water-column-sonar-processing
+Processing tool for converting L0 data to L1 and L2 as well as generating geospatial information
+
+## Setting up the Python Environment
+> Python 3.10.12
+
+# MacOS Pyenv Installation Instructions
+  1. Install pyenv (https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv)
+     1. ```brew update```
+     2. ```arch -arm64 brew install pyenv```
+     3. In ~/.bashrc add
+        1. ```export PYENV_ROOT="$HOME/.pyenv"```
+        2. ```export PATH="$PYENV_ROOT/bin:$PATH"```
+        3. ```eval "$(pyenv init -)"```
+     4. ```arch -arm64 brew install openssl readline sqlite3 xz zlib tcl-tk```
+  2. Install pyenv-virtualenv (https://github.com/pyenv/pyenv-virtualenv)
+     1. ```arch -arm64 brew install pyenv-virtualenv```
+     2. In ~/.bashrc add
+         1. ```eval "$(pyenv virtualenv-init -)"```
+  3. Open a new terminal
+  4. Install Python version
+     1. ```env PYTHON_CONFIGURE_OPTS="--enable-shared"```
+     2. ```env CONFIGURE_OPTS='--enable-optimizations' arch -arm64 pyenv install 3.10.12```
+  5. Create virtual env (to delete 'pyenv uninstall 3.10.12/water-column-sonar-processing')
+     1. ```pyenv virtualenv 3.10.12 water-column-sonar-processing```
+  6. Set local version of python (if not done already)
+     1. change directory to root of project
+     2. ```pyenv local 3.10.12 water-column-sonar-processing```
+     3. ```pyenv activate water-column-sonar-processing```
+
+## Setting up IntelliJ
+
+  1. Install the IntelliJ Python plugin
+  2. Set up pyenv
+     1. File -> Project Structure or CMD + ;
+     2. SDKs -> + -> Add Python SDK -> Virtual Environment
+     3. Select Existing Environment
+     4. Choose ~/.pyenv/versions/mocking_aws/bin/python
+  3. Set up Python Facet (not sure if this is required)
+     1. File -> Project Structure or CMD + ;
+     2. Facets -> + -> Python 
+     3. Set interpreter 
+
+## Installing Dependencies
+
+  1. Add dependencies with versions to requirements.txt
+  2. ```pip install --upgrade pip && pip install -r requirements_dev.txt```
+
+
+## Pytest
+```commandline
+pytest --disable-warnings
+```
+
+## Instructions
+Following this tutorial:
+https://packaging.python.org/en/latest/tutorials/packaging-projects/
+
+## To Publish To TEST
+```commandline
+python -m build
+python -m twine upload --repository testpypi dist/*
+pytho -m pip install --index-url https://test.pypi.org/simple/ hello-pypi-rudy-klucik
+python
+```
+```python
+from water-column-sonar-processing import ZarrManager
+example.add_one(2)
+```
+
+## To Publish To PROD
+```commandline
+python -m build
+python -m twine upload --repository pypi dist/*
+```

water_column_sonar_processing-0.0.1/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "water-column-sonar-processing"
+version = "0.0.1"
+authors = [
+  { name="Rudy Klucik", email="rudy.klucik@noaa.gov" },
+]
+description = "A processing tool for water column sonar data."
+readme = "README.md"
+requires-python = ">=3.10"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+
+[project.urls]
+Homepage = "https://github.com/CI-CMG/water-column-sonar-processing"
+Issues = "https://github.com/CI-CMG/water-column-sonar-processing/issues"

water_column_sonar_processing-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

water_column_sonar_processing-0.0.1/src/model/aws/dynamodb_manager.py

@@ -0,0 +1,149 @@
+import os
+import boto3
+import pandas as pd
+from boto3.dynamodb.types import TypeSerializer, TypeDeserializer
+
+
+#########################################################################
+class DynamoDBManager:
+    #####################################################################
+    def __init__(self):
+        self.__dynamodb_session = boto3.Session(
+            aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
+            aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
+            region_name=os.environ.get("AWS_REGION", default="us-east-1")
+        )
+        self.__dynamodb_resource = self.__dynamodb_session.resource(
+            service_name="dynamodb",
+        )
+        self.__dynamodb_client = self.__dynamodb_session.client(
+            service_name="dynamodb",
+        )
+        self.type_serializer = TypeSerializer()  # https://stackoverflow.com/a/46738251
+        self.type_deserializer = TypeDeserializer()
+
+    #####################################################################
+    ### defined in raw-to-zarr, not used
+    # def put_item(
+    #         self,
+    #         table_name,
+    #         item
+    # ):
+    #     response = boto3.Session().client(service_name='dynamodb').put_item(TableName=table_name, Item=item)
+    #     status_code = response['ResponseMetadata']['HTTPStatusCode']
+    #     assert (status_code == 200), "Problem, unable to update dynamodb table."
+
+    #####################################################################
+    def create_table(
+            self,
+            table_name,
+            key_schema,
+            attribute_definitions,
+    ):
+        self.__dynamodb_client.create_table(
+            AttributeDefinitions=attribute_definitions,
+            TableName=table_name,
+            KeySchema=key_schema,
+            BillingMode="PAY_PER_REQUEST",  # "PROVISIONED",
+            # ProvisionedThroughput={
+            #     'ReadCapacityUnits': 1_000,
+            #     'WriteCapacityUnits': 1_000
+            # }
+        )
+
+    #####################################################################
+    def get_item(
+            self,
+            table_name,
+            key
+    ):
+        response = self.__dynamodb_client.get_item(TableName=table_name, Key=key)
+        item = None
+        if response['ResponseMetadata']['HTTPStatusCode'] == 200:
+            if 'Item' in response:
+                item = response['Item']
+        return item
+
+    #####################################################################
+    def update_item(
+            self,
+            table_name,
+            key,
+            expression_attribute_names,
+            expression_attribute_values,
+            update_expression
+    ):
+        response = self.__dynamodb_client.update_item(
+            TableName=table_name,
+            Key=key,
+            ExpressionAttributeNames=expression_attribute_names,
+            ExpressionAttributeValues=expression_attribute_values,
+            UpdateExpression=update_expression
+        )
+        status_code = response['ResponseMetadata']['HTTPStatusCode']
+        # TODO: change to exception
+        assert (status_code == 200), "Problem, unable to update dynamodb table."
+
+    #####################################################################
+    def get_table_as_df(
+            self,
+            ship_name,
+            cruise_name,
+            sensor_name,
+            table_name,
+    ):
+        expression_attribute_values = {
+            ':cr': {'S': cruise_name},
+            ':se': {'S': sensor_name},
+            ':sh': {'S': ship_name},
+        }
+
+        filter_expression = 'CRUISE_NAME = :cr and SENSOR_NAME = :se and SHIP_NAME = :sh'
+        response = self.__dynamodb_client.scan(
+            TableName=table_name,
+            Select='ALL_ATTRIBUTES',
+            ExpressionAttributeValues=expression_attribute_values,
+            FilterExpression=filter_expression,
+        )
+        # Note: table.scan() has 1 MB limit on results so pagination is used
+        data = response['Items']
+
+        while 'LastEvaluatedKey' in response:
+            response = self.__dynamodb_client.scan(
+                TableName=table_name,
+                Select='ALL_ATTRIBUTES',
+                ExpressionAttributeValues=expression_attribute_values,
+                FilterExpression=filter_expression,
+                ExclusiveStartKey=response['LastEvaluatedKey']
+            )
+            data.extend(response['Items'])
+
+        deserializer = self.type_deserializer
+        df = pd.DataFrame([deserializer.deserialize({"M": i}) for i in data])
+
+        return df.sort_values(by='START_TIME', ignore_index=True)
+
+    #####################################################################
+    # is this used?
+    def get_table_item(
+            self,
+            table_name,
+            key,
+    ):
+        # a bit more high level, uses resource to get table item
+        table = self.__dynamodb_resource.Table(table_name)
+        response = table.get_item(
+            Key=key
+        )
+        return response
+
+    #####################################################################
+    # TODO: add helper method to delete the data
+    def delete_cruise(
+            self,
+            table_name,
+            cruise_name,
+    ):
+        pass
+
+#########################################################################