PyOPIA 2.16.14__tar.gz → 2.16.16__tar.gz

{pyopia-2.16.14 → pyopia-2.16.16}/.gitignore

@@ -267,7 +267,7 @@ poetry.lock
 /notebooks/oil_silcam_images
 header.tfl.txt
 *.tiff
-notebooks/__MACOSX/*
+**/__MACOSX/*
 notebooks/silcam240822.keras
 model/*
 notebooks/model/*

{pyopia-2.16.14 → pyopia-2.16.16}/PKG-INFO

@@ -1,10 +1,11 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: PyOPIA
-Version: 2.16.14
+Version: 2.16.16
 Summary: A Python Ocean Particle Image Analysis toolbox.
 Project-URL: Repository, https://github.com/sintef/pyopia
 Project-URL: Documentation, https://pyopia.readthedocs.io
 Author-email: Emlyn Davies <emlyn.davies@sintef.no>, Alex Nimmo Smith <alex@nimmosmith.co.uk>
+License-Expression: BSD-3-Clause
 License-File: LICENSE
 Keywords: Imaging,Measurement,Ocean,Particles,Size distribution
 Requires-Python: ~=3.12.0
@@ -27,6 +28,8 @@ Requires-Dist: pandas[computation]<3,>=2.1.1
 Requires-Dist: poetry-version-plugin<0.3,>=0.2.0
 Requires-Dist: pydantic>=2.11.7
 Requires-Dist: pytest-error-for-skips<3,>=2.0.2
+Requires-Dist: pytest-rerunfailures<15,>=14.0
+Requires-Dist: pytest-timeout<3,>=2.3.1
 Requires-Dist: pytest>=7.2.0
 Requires-Dist: readthedocs-sphinx-search<0.4,>=0.3.2
 Requires-Dist: scikit-image<0.25,>=0.24.0
@@ -55,10 +58,12 @@ PyOPIA
 
 A Python Ocean Particle Image Analysis toolbox
 
+PyOPIA processes images of particles suspended in water (e.g. from SilCam, holographic, or UVP imaging systems) into particle size, shape, and concentration statistics.
+
 # Quick tryout of PyOPIA
 
 1) Install [uv](https://docs.astral.sh/uv/getting-started/installation)
-2) Initialize PyOPIA project with a small example image dataset and run processing
+2) Initialize a PyOPIA project with example data (`--example-data` downloads a small example image dataset and generates a matching `config.toml`), then run processing:
 ```bash
 uvx --python 3.12 --from pyopia[classification] pyopia init-project pyopiatest --example-data
 ```
@@ -76,19 +81,15 @@ uvx --python 3.12 --from pyopia[classification] pyopia merge-mfdata processed
 ```
 
 ```bash
-uvx --python 3.12 --from pyopia[classification] pyopia make-montage processed\pyopiatest-STATS.nc
-```
-5) Visualise the monatge of all processed singular particle images in one
-```bash
-montage.png
+uvx --python 3.12 --from pyopia[classification] pyopia make-montage processed/pyopiatest-STATS.nc
 ```
-Will show you a montage of all the processed particle images in one.
+5) This creates `montage.png` in the current folder - open it to see a single image made up of all the processed particle images.
 
 See the documentation for more information on how to install and use PyOPIA.
 
 # Running with Docker
 
-A prebuilt container image is published to GitHub Container Registry on every release, for users who prefer not to install PyOPIA's dependencies directly.
+A prebuilt container image is published to GitHub Container Registry on every release. Docker is worth reaching for if you'd rather not install PyOPIA's dependencies directly: it avoids the install overhead of heavier optional dependencies (e.g. TensorFlow/PyTorch for classification), guarantees a consistent, reproducible environment regardless of your host OS, and is well suited to running on servers or HPC systems.
 
 ## One-off invocation
 
@@ -161,41 +162,57 @@ PYOPIA_CONFIG=my_run.toml docker compose run --rm pyopia
 [pyopia.readthedocs.io](https://pyopia.readthedocs.io)
 # Current status:
 
-- Under development. See/regester issues, [here](https://github.com/SINTEF/pyopia/issues)
+- Under development. See/register issues, [here](https://github.com/SINTEF/pyopia/issues)
 
 ----
-# Development targets for PyOpia:
+# Releases
 
-1) Allow nonfamiliar users to install and use PyOpia, and to contribute & commit code changes
-2) Not hardware specific
-3) Smaller dependency list than PySilCam -Eventual optional dependencies (e.g. for classification)
-4) Can be imported by pysilcam or other hardware-specific tools
-5) Work on a single-image basis (...primarily, with options for multiprocess to be considered later)
-6) No use of settings/config files within the core code - pass arguments directly. Eventual use of settings/config files should be handled by high-level wrappers that provide arguments to functions.
-7) Github workflows
-8) Tests
+Releases are cut when a finished, documented, and tested piece of work is ready to be used, rather than on a fixed schedule. If a feature has landed on `main` and meets the Contributions guidelines below, it should go out in the next release rather than sit unreleased. We aim to keep the Docker image (see Running with Docker above) in sync with each release, publishing it as part of the release process rather than as a separate manual step.
 
-Normal functions within PyOpia should:
+----
+# Design principles
 
-1) take inputs
-2) return new outputs
-3) don't modify state of input
-4) minimum possible disc IO during processing
+- PyOPIA is instrument-agnostic at its core: SilCam, holographic, and UVP support are all built as pluggable instrument modules on top of a shared `Pipeline`.
+- Processing is config-driven: a `Pipeline` is built from a TOML/dict `settings` object describing an ordered list of steps, each mapping to a Python class. Steps update a shared `data` dict as the pipeline runs.
+- Heavy, optional dependencies (e.g. TensorFlow/PyTorch for classification) are kept out of the core install and available via extras (`pyopia[classification]`, `pyopia[classification-torch]`).
+- Multi-file, multi-core processing is supported via chunked/parallel processing (`pyopia process --num-chunks`).
 
 ## Contributions
 
 We welcome additions and improvements to the code! We request that you follow a few guidelines. These are in place to make sure the code improves over time.
 
 1. All code changes must be submitted as pull requests, either from a branch or a fork.
-2. Good documentation of the code is needed for PyOpia to succeed and so please include up-to-date docstrings as you make changes, so that the auto-build on readthedocs is complete and useful for users. (A version of the new docs will complie when you make a pull request and a link to this can be found in the pull request checks)
+2. Good documentation of the code is needed for PyOPIA to succeed and so please include up-to-date docstrings as you make changes, so that the auto-build on readthedocs is complete and useful for users. (A version of the new docs will compile when you make a pull request and a link to this can be found in the pull request checks)
 3. All pull requests are required to pass all tests before merging. Please do not disable or remove tests just to make your branch pass the pull request.
-4. All pull requests must be reviewed by a person. The benefits from code review are plenty, but we like to emphasise that code reviews help spreading the awarenes of code changes. Please note that code reviews should be a pleasant experience, so be plesant, polite and remember that there is a human being with good intentions on the other side of the screen.
+4. All pull requests must be reviewed by a person. The benefits from code review are plenty, but we like to emphasise that code reviews help spreading the awareness of code changes. Please note that code reviews should be a pleasant experience, so be pleasant, polite and remember that there is a human being with good intentions on the other side of the screen.
 5. All contributions are linted with flake8. We recommend that you run flake8 on your code while developing to fix any issues as you go. We recommend using autopep8 to autoformat your Python code (but please check the code behaviour is not affected by autoformatting before pushing). This makes flake8 happy, and makes it easier for us all to maintain a consistent and readable code base.
 
+### Using AI tools
+
+AI coding tools (e.g. Claude, Copilot, ChatGPT) are welcome as an aid to writing PyOPIA contributions. They are tools, not authors: whoever submits a pull request is responsible for the code in it, regardless of how much of it an AI tool helped produce. Please review, understand, and test any AI-assisted changes yourself before submitting them - the same guidelines above (documentation, tests, review, flake8) apply either way.
+
 ## Docstrings
 
 Use the NumPy style in docstrings. See style guide [here](https://numpydoc.readthedocs.io/en/latest/format.html#documenting-classes)
 
+## Testing
+
+PyOPIA's test suite lives in `pyopia/tests/` and runs via `pytest` (see `uv run pytest` below). A few things are useful to know before running or adding to it:
+
+**Markers**: some tests are tagged with pytest markers to indicate how expensive they are.
+
+- `@pytest.mark.slow` - tests that do real network downloads and/or real model inference (e.g. downloading the example classifier model and running real predictions on it). These run in routine CI, but you can skip them for a fast local feedback loop:
+  ```bash
+  uv run pytest -m "not slow"
+  ```
+Unmarked tests are fast and have no external dependencies; they always run.
+
+**Shared fixtures**: tests that need real example data (an example image, the trained classifier model, the classifier training database, an example hologram) get it from session-scoped fixtures defined in `pyopia/tests/conftest.py`, rather than each downloading their own copy. The download happens once per test run and is shared across every test file that needs it.
+
+**Notebooks**: `pyopia/tests/test_notebooks.py` executes the notebooks in `notebooks/` and `docs/notebooks/` to check they still run against the current codebase. Each notebook is its own parametrized test (`test_notebook[<name>.ipynb]`), tagged `slow` as above where relevant. Not every notebook is included - a couple depend on state produced by another notebook, or by a user's own prior processing run, and would fail if executed standalone; see the comments in `test_notebooks.py` for which ones and why. These tests also get a longer timeout (1800s) and up to 2 automatic retries on failure, since running a Jupyter kernel via nbconvert has shown real, platform-specific flakiness on macOS CI runners rather than a reproducible bug.
+
+Please do not disable or remove tests just to make a pull request pass - see Contributions guideline 3 above.
+
 # Installing
 
 ## For users
@@ -205,7 +222,7 @@ Users are expected to be familiar with Python. Please refer to the recommended i
 ## For developers from source
 
 
-Install (uv)[https://docs.astral.sh/uv/getting-started/installation/]
+Install [uv](https://docs.astral.sh/uv/getting-started/installation/)
 
 1. Navigate to the folder where you want to install pyopia using the 'cd' command.
 
@@ -222,10 +239,10 @@ For the next steps, you need to be located in the PyOPIA root directory that con
 2. Install all requirements with
 
 ```bash
-uv sync --all-extras
+uv sync --all-extras --dev
 ```
 
-3. (optional) Run local tests:
+3. (optional) Run local tests (see the Testing section above for markers and how to run a fast subset):
 
 ```bash
 uv run pytest
@@ -241,13 +258,12 @@ The version number of PyOPIA is split into three sections: MAJOR.MINOR.PATCH
 
 ## Build docs locally
 
-```
-sphinx-apidoc -f -o docs/source docs/build --separate
-
-sphinx-build -b html ./docs/ ./docs/build
+```bash
+uv sync --extra classification --group docs
+uv run sphinx-build -b html docs/ docs/_build/html
 ```
 
 ----
 # License
 
-PyOpia is licensed under the BSD3 license. See LICENSE. All contributors should be recognised & aknowledged.
+PyOPIA is licensed under the BSD3 license. See LICENSE. All contributors should be recognised & acknowledged.

{pyopia-2.16.14 → pyopia-2.16.16}/README.md

@@ -3,10 +3,12 @@ PyOPIA
 
 A Python Ocean Particle Image Analysis toolbox
 
+PyOPIA processes images of particles suspended in water (e.g. from SilCam, holographic, or UVP imaging systems) into particle size, shape, and concentration statistics.
+
 # Quick tryout of PyOPIA
 
 1) Install [uv](https://docs.astral.sh/uv/getting-started/installation)
-2) Initialize PyOPIA project with a small example image dataset and run processing
+2) Initialize a PyOPIA project with example data (`--example-data` downloads a small example image dataset and generates a matching `config.toml`), then run processing:
 ```bash
 uvx --python 3.12 --from pyopia[classification] pyopia init-project pyopiatest --example-data
 ```
@@ -24,19 +26,15 @@ uvx --python 3.12 --from pyopia[classification] pyopia merge-mfdata processed
 ```
 
 ```bash
-uvx --python 3.12 --from pyopia[classification] pyopia make-montage processed\pyopiatest-STATS.nc
-```
-5) Visualise the monatge of all processed singular particle images in one
-```bash
-montage.png
+uvx --python 3.12 --from pyopia[classification] pyopia make-montage processed/pyopiatest-STATS.nc
 ```
-Will show you a montage of all the processed particle images in one.
+5) This creates `montage.png` in the current folder - open it to see a single image made up of all the processed particle images.
 
 See the documentation for more information on how to install and use PyOPIA.
 
 # Running with Docker
 
-A prebuilt container image is published to GitHub Container Registry on every release, for users who prefer not to install PyOPIA's dependencies directly.
+A prebuilt container image is published to GitHub Container Registry on every release. Docker is worth reaching for if you'd rather not install PyOPIA's dependencies directly: it avoids the install overhead of heavier optional dependencies (e.g. TensorFlow/PyTorch for classification), guarantees a consistent, reproducible environment regardless of your host OS, and is well suited to running on servers or HPC systems.
 
 ## One-off invocation
 
@@ -109,41 +107,57 @@ PYOPIA_CONFIG=my_run.toml docker compose run --rm pyopia
 [pyopia.readthedocs.io](https://pyopia.readthedocs.io)
 # Current status:
 
-- Under development. See/regester issues, [here](https://github.com/SINTEF/pyopia/issues)
+- Under development. See/register issues, [here](https://github.com/SINTEF/pyopia/issues)
 
 ----
-# Development targets for PyOpia:
+# Releases
 
-1) Allow nonfamiliar users to install and use PyOpia, and to contribute & commit code changes
-2) Not hardware specific
-3) Smaller dependency list than PySilCam -Eventual optional dependencies (e.g. for classification)
-4) Can be imported by pysilcam or other hardware-specific tools
-5) Work on a single-image basis (...primarily, with options for multiprocess to be considered later)
-6) No use of settings/config files within the core code - pass arguments directly. Eventual use of settings/config files should be handled by high-level wrappers that provide arguments to functions.
-7) Github workflows
-8) Tests
+Releases are cut when a finished, documented, and tested piece of work is ready to be used, rather than on a fixed schedule. If a feature has landed on `main` and meets the Contributions guidelines below, it should go out in the next release rather than sit unreleased. We aim to keep the Docker image (see Running with Docker above) in sync with each release, publishing it as part of the release process rather than as a separate manual step.
 
-Normal functions within PyOpia should:
+----
+# Design principles
 
-1) take inputs
-2) return new outputs
-3) don't modify state of input
-4) minimum possible disc IO during processing
+- PyOPIA is instrument-agnostic at its core: SilCam, holographic, and UVP support are all built as pluggable instrument modules on top of a shared `Pipeline`.
+- Processing is config-driven: a `Pipeline` is built from a TOML/dict `settings` object describing an ordered list of steps, each mapping to a Python class. Steps update a shared `data` dict as the pipeline runs.
+- Heavy, optional dependencies (e.g. TensorFlow/PyTorch for classification) are kept out of the core install and available via extras (`pyopia[classification]`, `pyopia[classification-torch]`).
+- Multi-file, multi-core processing is supported via chunked/parallel processing (`pyopia process --num-chunks`).
 
 ## Contributions
 
 We welcome additions and improvements to the code! We request that you follow a few guidelines. These are in place to make sure the code improves over time.
 
 1. All code changes must be submitted as pull requests, either from a branch or a fork.
-2. Good documentation of the code is needed for PyOpia to succeed and so please include up-to-date docstrings as you make changes, so that the auto-build on readthedocs is complete and useful for users. (A version of the new docs will complie when you make a pull request and a link to this can be found in the pull request checks)
+2. Good documentation of the code is needed for PyOPIA to succeed and so please include up-to-date docstrings as you make changes, so that the auto-build on readthedocs is complete and useful for users. (A version of the new docs will compile when you make a pull request and a link to this can be found in the pull request checks)
 3. All pull requests are required to pass all tests before merging. Please do not disable or remove tests just to make your branch pass the pull request.
-4. All pull requests must be reviewed by a person. The benefits from code review are plenty, but we like to emphasise that code reviews help spreading the awarenes of code changes. Please note that code reviews should be a pleasant experience, so be plesant, polite and remember that there is a human being with good intentions on the other side of the screen.
+4. All pull requests must be reviewed by a person. The benefits from code review are plenty, but we like to emphasise that code reviews help spreading the awareness of code changes. Please note that code reviews should be a pleasant experience, so be pleasant, polite and remember that there is a human being with good intentions on the other side of the screen.
 5. All contributions are linted with flake8. We recommend that you run flake8 on your code while developing to fix any issues as you go. We recommend using autopep8 to autoformat your Python code (but please check the code behaviour is not affected by autoformatting before pushing). This makes flake8 happy, and makes it easier for us all to maintain a consistent and readable code base.
 
+### Using AI tools
+
+AI coding tools (e.g. Claude, Copilot, ChatGPT) are welcome as an aid to writing PyOPIA contributions. They are tools, not authors: whoever submits a pull request is responsible for the code in it, regardless of how much of it an AI tool helped produce. Please review, understand, and test any AI-assisted changes yourself before submitting them - the same guidelines above (documentation, tests, review, flake8) apply either way.
+
 ## Docstrings
 
 Use the NumPy style in docstrings. See style guide [here](https://numpydoc.readthedocs.io/en/latest/format.html#documenting-classes)
 
+## Testing
+
+PyOPIA's test suite lives in `pyopia/tests/` and runs via `pytest` (see `uv run pytest` below). A few things are useful to know before running or adding to it:
+
+**Markers**: some tests are tagged with pytest markers to indicate how expensive they are.
+
+- `@pytest.mark.slow` - tests that do real network downloads and/or real model inference (e.g. downloading the example classifier model and running real predictions on it). These run in routine CI, but you can skip them for a fast local feedback loop:
+  ```bash
+  uv run pytest -m "not slow"
+  ```
+Unmarked tests are fast and have no external dependencies; they always run.
+
+**Shared fixtures**: tests that need real example data (an example image, the trained classifier model, the classifier training database, an example hologram) get it from session-scoped fixtures defined in `pyopia/tests/conftest.py`, rather than each downloading their own copy. The download happens once per test run and is shared across every test file that needs it.
+
+**Notebooks**: `pyopia/tests/test_notebooks.py` executes the notebooks in `notebooks/` and `docs/notebooks/` to check they still run against the current codebase. Each notebook is its own parametrized test (`test_notebook[<name>.ipynb]`), tagged `slow` as above where relevant. Not every notebook is included - a couple depend on state produced by another notebook, or by a user's own prior processing run, and would fail if executed standalone; see the comments in `test_notebooks.py` for which ones and why. These tests also get a longer timeout (1800s) and up to 2 automatic retries on failure, since running a Jupyter kernel via nbconvert has shown real, platform-specific flakiness on macOS CI runners rather than a reproducible bug.
+
+Please do not disable or remove tests just to make a pull request pass - see Contributions guideline 3 above.
+
 # Installing
 
 ## For users
@@ -153,7 +167,7 @@ Users are expected to be familiar with Python. Please refer to the recommended i
 ## For developers from source
 
 
-Install (uv)[https://docs.astral.sh/uv/getting-started/installation/]
+Install [uv](https://docs.astral.sh/uv/getting-started/installation/)
 
 1. Navigate to the folder where you want to install pyopia using the 'cd' command.
 
@@ -170,10 +184,10 @@ For the next steps, you need to be located in the PyOPIA root directory that con
 2. Install all requirements with
 
 ```bash
-uv sync --all-extras
+uv sync --all-extras --dev
 ```
 
-3. (optional) Run local tests:
+3. (optional) Run local tests (see the Testing section above for markers and how to run a fast subset):
 
 ```bash
 uv run pytest
@@ -189,13 +203,12 @@ The version number of PyOPIA is split into three sections: MAJOR.MINOR.PATCH
 
 ## Build docs locally
 
-```
-sphinx-apidoc -f -o docs/source docs/build --separate
-
-sphinx-build -b html ./docs/ ./docs/build
+```bash
+uv sync --extra classification --group docs
+uv run sphinx-build -b html docs/ docs/_build/html
 ```
 
 ----
 # License
 
-PyOpia is licensed under the BSD3 license. See LICENSE. All contributors should be recognised & aknowledged.
+PyOPIA is licensed under the BSD3 license. See LICENSE. All contributors should be recognised & acknowledged.

pyopia-2.16.16/pyopia/__init__.py

@@ -0,0 +1 @@
+__version__ = "2.16.16"

{pyopia-2.16.14 → pyopia-2.16.16}/pyopia/auxillarydata.py

@@ -4,6 +4,11 @@ import xarray as xr
 
 logger = logging.getLogger()
 
+
+class AuxillaryDataError(Exception):
+    """Raised when an auxiliary data file exists but cannot be parsed into the expected format."""
+
+
 AUXILLARY_DATA_FILE_TEMPLATE = """% COMMENT LINE: PLEASE UPDATE THIS FILE WITH PROJECT RELEVANT DATA. EACH COLUMN WILL BECOME A NETCDF VARIABLE.
 % COMMENT LINE: ONE LINE PER MEASUREMENT, TIME IS INTERPOLATED TO IMAGE DATA TIMES IN PYOPIA. FOLLOWING LINES ARE UNITS, DESCRIPTION AND VARIABLE NAME.
 ,metres,degC
@@ -50,17 +55,19 @@ class AuxillaryData:
     def __init__(self, auxillary_data_path=None):
         self.auxillary_data_path = auxillary_data_path
 
-        # Create empty dataframe for cases where no file was specified, or an error occured reading it
+        # Create empty dataframe for cases where no file was specified
         self.auxillary_data = pd.DataFrame(index=pd.Index([], name="time")).to_xarray()
         if auxillary_data_path is not None:
             try:
                 self.auxillary_data = self.load_auxillary_data(auxillary_data_path)
-            except RuntimeError as e:
-                print(f"Failed to load auxillary data from file: {self.auxillary_data}")
-                logging.error(
-                    f"Failed to load auxillary data from file: {self.auxillary_data}"
-                )
-                logging.error(e)
+            except Exception as e:
+                # Re-raise as a single, specific error type that names the file and the
+                # underlying cause, so callers can recognise this failure mode (the same
+                # auxiliary data file applies to every image, so it is not worth retrying)
+                # and report something more useful than a bare KeyError/ValueError.
+                raise AuxillaryDataError(
+                    f"Failed to load auxiliary data from '{auxillary_data_path}': {e}"
+                ) from e
 
     def load_auxillary_data(self, auxillary_data_path):
         """Load and format uxillary data from .csv file"""