tonik 0.0.1__tar.gz → 0.0.3__tar.gz

tonik-0.0.3/.devcontainer/devcontainer.json

@@ -0,0 +1,22 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the
+// README at: https://github.com/devcontainers/templates/tree/main/src/python
+{
+	"name": "Python 3",
+	// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
+	"image": "mcr.microsoft.com/devcontainers/python:1-3.10-bullseye"
+
+	// Features to add to the dev container. More info: https://containers.dev/features.
+	// "features": {},
+
+	// Use 'forwardPorts' to make a list of ports inside the container available locally.
+	// "forwardPorts": [],
+
+	// Use 'postCreateCommand' to run commands after the container is created.
+	// "postCreateCommand": "pip3 install --user -r requirements.txt",
+
+	// Configure tool-specific properties.
+	// "customizations": {},
+
+	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
+	// "remoteUser": "root"
+}

tonik-0.0.3/HOW_TO_RELEASE.md

@@ -0,0 +1,51 @@
+# How to issue a new zizou release
+
+## Pypi
+Install the build system
+```
+python3 -m pip install --upgrade build
+```
+Install `twine`:
+```
+python3 -m pip install --upgrade twine
+```
+Make sure the version number is incremented in the `pyproject.toml` file.
+
+Run the build to create two new files under `dist`. One is the wheel and the other the packaged source code:
+```
+python3 -m build
+```
+Upload to pypi:
+```
+python3 -m twine upload dist/*
+```
+When prompted for the username enter `__token__` and paste in your pypi token as the password.
+
+## Documentation
+Install the [mkdocs](https://www.mkdocs.org/) package and the [mkdocstrings](https://mkdocstrings.github.io/) plugin:
+```
+python3 -m pip install --upgrade mkdocs "mkdocstrings[python]"
+```
+
+To view the documentation locally run:
+```
+mkdocs serve
+```
+To generate documentation run:
+```
+mkdocs build
+```
+from the project root directory. This creates a directory called `site` containing all the necessary files to host the documentation.
+Please don't add `site` to version control. If it is the first time you built the documentation, run the following:
+
+```
+mv site ../tonik.github.io
+cd ../tonik.github.io
+git init
+git add .
+git commit -m "update documentation"
+git branch -m main
+git add origin git@github.com:tsc-tools/tonik.github.io.git
+git push -u --force origin main
+```
+[Github pages](https://pages.github.com/) will then publish the documentation under https://tsc-tools.github.io/tonik.github.io

{tonik-0.0.1 → tonik-0.0.3}/PKG-INFO

@@ -1,8 +1,8 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.3
 Name: tonik
-Version: 0.0.1
+Version: 0.0.3
 Summary: A collection of tools to integrate with GNS Science's time series classification platform.
-Project-URL: Homepage, https://github.com/tsc-tools/tonik
+Project-URL: Homepage, https://tsc-tools.github.io/tonik.github.io
 Project-URL: Issues, https://github.com/tsc-tools/tonik/issues
 Author-email: Yannik Behr <y.behr@gns.cri.nz>, Christof Mueller <c.mueller@gns.cri.nz>
 License-File: LICENSE
@@ -10,12 +10,17 @@ Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
 Requires-Python: >=3.7
-Requires-Dist: h5netcdf
-Requires-Dist: h5py
-Requires-Dist: netcdf4
-Requires-Dist: pandas
-Requires-Dist: pytest
-Requires-Dist: xarray
+Requires-Dist: datashader>=0.14
+Requires-Dist: fastapi>=0.95
+Requires-Dist: h5netcdf>=1.1
+Requires-Dist: h5py>=3.8
+Requires-Dist: netcdf4>=1.6
+Requires-Dist: pandas>=2.0
+Requires-Dist: python-json-logger>=2.0
+Requires-Dist: uvicorn[standard]>=0.22
+Requires-Dist: xarray>=2023.4
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
 Description-Content-Type: text/markdown
 
 # Time series classification tools

tonik-0.0.3/docs/user_guide.md

@@ -0,0 +1,65 @@
+## Examples on how to store and retrieve data using lockerroom
+
+```python
+from datetime import datetime
+import os
+import numpy as np
+import pandas as pd
+import xarray as xr
+from tonik import StorageGroup 
+```
+
+A locker room consists of many lockers. Each locker stores data for a single sensor and the locker room groups sensors by, for example, experiment or geographic location. We will start by generating fake spectrogram data.
+
+```python
+dates = pd.date_range("2024-01-02", freq='10min', periods=288)
+data = np.abs(np.cumsum(np.random.normal(0, 8., len(dates))))
+data = np.tile(data, (10, 1))
+freqs = np.arange(10)
+xrd = xr.Dataset({'spectrogram': xr.DataArray(data, coords=[freqs, dates],
+                                              dims=['frequency', 'datetime'])})
+xrd['spectrogram'].plot()
+```
+
+![1st spectrogram](img/spectrogram1.png)
+
+Now we will store the data under two different sites of the same experiment.
+
+```python
+g = StorageGroup('test_experiment', rootdir='/tmp')
+st1 = g.get_store(site='MDR1', sensor='00', channel='HHZ')
+st2 = g.get_store(site='MDR2', sensor='00', channel='HHZ')
+st1.save(xrd)
+st2.save(xrd)
+g
+```
+
+![lockerroom output](img/storagegroup_output.png)
+
+Next we want to retrieve the data we just saved. Before retrieving data we have to set the timespan over which we want to retrieve data.
+
+```python
+g.starttime = datetime(2024, 1, 2, 18, 0, 0)
+g.endtime = datetime(2024, 1, 3, 6, 0, 0)
+st = g.get_store('MDR2', '00', 'HHZ')
+st('spectrogram').plot()
+```
+
+![2nd spectrogram](img/spectrogram2.png)
+
+Tonik comes with an API to access the stored data from other applications. To test the API, first start it in a terminal:
+```
+tonik_api --rootdir /tmp
+```
+
+Now you can access the same feature through the API:
+
+```python
+url = "http://localhost:8003/feature?"
+url += f"group={g.name}&site=MDR2&sensor=00&channel=HHZ&name=spectrogram"
+url += f"&starttime={g.starttime.isoformat()}&endtime={g.endtime.isoformat()}" 
+spec = pd.read_csv(url, parse_dates=True, index_col=[0, 1], date_format='ISO8601').to_xarray()
+spec.transpose('freqs', 'dates')['feature'].plot()
+```
+
+![2nd spectrogram](img/spectrogram2.png)

tonik-0.0.3/examples/tonik_example.ipynb

@@ -0,0 +1,131 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Examples on how to store and retrieve data using lockerroom"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datetime import datetime\n",
+    "import os\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "import xarray as xr\n",
+    "from tonik import StorageGroup "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A locker room consists of many lockers. Each locker stores data for a single sensor and the locker room groups sensors by, for example, experiment or geographic location. We will start by generating fake spectrogram data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dates = pd.date_range(\"2024-01-02\", freq='10min', periods=288)\n",
+    "data = np.abs(np.cumsum(np.random.normal(0, 8., len(dates))))\n",
+    "data = np.tile(data, (10, 1))\n",
+    "freqs = np.arange(10)\n",
+    "xrd = xr.Dataset({'spectrogram': xr.DataArray(data, coords=[freqs, dates],\n",
+    "                                              dims=['frequency', 'datetime'])})\n",
+    "fig = xrd['spectrogram'].plot()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we will store the data under two different sites of the same experiment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "g = StorageGroup('test_experiment', rootdir='/tmp')\n",
+    "st1 = g.get_store(site='MDR1', sensor='00', channel='HHZ')\n",
+    "st2 = g.get_store(site='MDR2', sensor='00', channel='HHZ')\n",
+    "st1.save(xrd)\n",
+    "st2.save(xrd)\n",
+    "g"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next we want to retrieve the data we just saved. Before retrieving data we have to set the timespan over which we want to retrieve data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "g.starttime = datetime(2024, 1, 2, 18, 0, 0)\n",
+    "g.endtime = datetime(2024, 1, 3, 6, 0, 0)\n",
+    "st = g.get_store('MDR2', '00', 'HHZ')\n",
+    "st('spectrogram').plot()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Start the API server by running the following command in the terminal\n",
+    "# tonik_api --rootdir examples"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = \"http://localhost:8003/feature?\"\n",
+    "url += f\"group={g.name}&site=MDR2&sensor=00&channel=HHZ&name=spectrogram\"\n",
+    "url += f\"&starttime={g.starttime.isoformat()}&endtime={g.endtime.isoformat()}\" \n",
+    "spec = pd.read_csv(url, parse_dates=True, index_col=[0, 1], date_format='ISO8601').to_xarray()\n",
+    "spec.transpose('freqs', 'dates')['feature'].plot()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.14"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

tonik-0.0.3/mkdocs.yml

@@ -0,0 +1,5 @@
+site_name: Tonik 
+nav:
+  - Home: index.md
+  - User Guide: user_guide.md
+theme: readthedocs

{tonik-0.0.1 → tonik-0.0.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "tonik"
-version = "0.0.1"
+version = "0.0.3"
 authors = [
   { name="Yannik Behr", email="y.behr@gns.cri.nz" },
   { name="Christof Mueller", email="c.mueller@gns.cri.nz" }
@@ -19,14 +19,23 @@ classifiers = [
     "Operating System :: OS Independent",
 ]
 dependencies = [
-    "h5py",
-    "xarray",
-    "pandas",
-    "netcdf4",
-    "h5netcdf",
-    "pytest"
+    "h5py>=3.8",
+    "datashader>=0.14",
+    "xarray>=2023.4",
+    "pandas>=2.0",
+    "netcdf4>=1.6",
+    "h5netcdf>=1.1",
+    "python-json-logger>=2.0",
+    "uvicorn[standard]>=0.22",
+    "fastapi>=0.95"
 ]
 
+[project.optional-dependencies]
+dev = ["pytest"]
+
 [project.urls]
-Homepage = "https://github.com/tsc-tools/tonik"
+Homepage = "https://tsc-tools.github.io/tonik.github.io"
 Issues = "https://github.com/tsc-tools/tonik/issues"
+
+[project.scripts]
+tonik_api = "tonik.api:main"

tonik-0.0.3/src/tonik/__init__.py

@@ -0,0 +1,23 @@
+import importlib
+from os import PathLike
+from typing import Optional
+
+from .storage import StorageGroup, Path 
+from .utils import generate_test_data
+
+
+def get_data(filename: Optional[PathLike] = None) -> str:
+    """Return path to tonik package.
+
+    Parameters
+    ----------
+    filename : Pathlike, default None
+        Append `filename` to returned path.
+
+    Returns
+    -------
+    pkgdir_path
+
+    """
+    f = importlib.resources.files(__package__)
+    return str(f) if filename is None else str(f / filename)

tonik-0.0.3/src/tonik/api.py

@@ -0,0 +1,276 @@
+from argparse import ArgumentParser
+from datetime import timedelta, datetime
+import logging
+import os
+
+from cftime import num2date, date2num
+import datashader as dsh
+import numpy as np
+import pandas as pd
+import uvicorn
+from fastapi import FastAPI, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import HTMLResponse, StreamingResponse
+from pydantic import BaseModel
+from typing import List
+
+from .storage import StorageGroup
+from . import get_data
+
+logger = logging.getLogger(__name__)
+
+                          
+class TonikAPI:
+
+    def __init__(self, rootdir) -> None:
+        self.rootdir = rootdir
+        self.app = FastAPI()
+
+        # -- allow any origin to query API
+        self.app.add_middleware(CORSMiddleware,
+                                allow_origins=["*"])
+
+        self.app.get("/", response_class=HTMLResponse)(self.root)
+        self.app.get("/feature")(self.feature)
+
+    async def root(self):
+        with open(get_data("package_data/index.html"), "r", encoding="utf-8") as file:
+            html_content = file.read()
+        return HTMLResponse(content=html_content, status_code=200)
+
+
+    def feature(self,
+                name: str='rsam',
+                group: str='Ruapehu',
+                site: str='MAVZ',
+                sensor: str='10',
+                channel: str='HHZ',
+                starttime: datetime=datetime.utcnow()-timedelta(days=30),
+                endtime: datetime=datetime.utcnow(),
+                resolution: str='full',
+                verticalres: int=10,
+                log: bool=True,
+                normalise: bool=False):
+        
+        _st = datetime.fromisoformat(str(starttime))
+        _st = _st.replace(tzinfo=None)
+        _et = datetime.fromisoformat(str(endtime))
+        _et = _et.replace(tzinfo=None)
+        g = StorageGroup(group, rootdir=self.rootdir,
+                        starttime=_st, endtime=_et)
+        c = g.get_store(site=site, sensor=sensor, channel=channel)
+        try:
+            feat = c(name)
+        except ValueError as e:
+            msg = f"Feature {name} not found in directory {l.sitedir}:"
+            msg += f"{e}"
+            raise HTTPException(status_code=404, detail=msg)
+        if len(feat.shape) > 1:
+            # assume first dimension is frequency
+            nfreqs = feat.shape[0]
+            dates = feat.coords[feat.dims[1]].values
+            if resolution != 'full':
+                freq, dates, spec = self.aggregate_feature(resolution, verticalres, feat, nfreqs, dates)
+            else:
+                spec = feat.values
+                freq = feat.coords[feat.dims[0]].values
+            vals = spec.ravel(order='C')
+            if log and feat.name != 'sonogram':
+                vals = 10*np.log10(vals)
+            if normalise:
+                vals = (vals - np.nanmin(vals))/(np.nanmax(vals) - np.nanmin(vals))
+            freqs = freq.repeat(dates.size)
+            dates = np.tile(dates, freq.size)
+            df = pd.DataFrame({'dates': dates, 'freqs': freqs, 'feature': vals})
+            output = df.to_csv(index=False,
+                            columns=['dates', 'freqs', 'feature'])
+        else:
+            df = pd.DataFrame(data=feat.to_pandas(), columns=[feat.name])
+            df['dates'] = df.index
+            try:
+                df = df.resample(str(float(resolution)/60000.0)+'T').mean()
+            except ValueError as e:
+                logger.warning(f"Cannot resample {feat.name} to {resolution}: e")
+            df.rename(columns={feat.name: 'feature'}, inplace=True)
+            output = df.to_csv(index=False, columns=['dates', 'feature'])
+        return StreamingResponse(iter([output]),
+                                media_type='text/csv',
+                                headers={"Content-Disposition":
+                                        "attachment;filename=<VUMT_feature>.csv",
+                                        'Content-Length': str(len(output))})
+
+
+    def aggregate_feature(self, resolution, verticalres, feat, nfreqs, dates):
+        resolution = np.timedelta64(pd.Timedelta(resolution), 'ms').astype(float)
+        ndays = np.timedelta64(dates[-1] - dates[0], 'ms').astype(float)
+        canvas_x =  int(ndays/resolution)
+        canvas_y = min(nfreqs, verticalres)
+        dates = date2num(dates.astype('datetime64[us]').astype(datetime),
+                                units='hours since 1970-01-01 00:00:00.0',
+                                calendar='gregorian')
+        feat = feat.assign_coords({'datetime': dates})
+        cvs = dsh.Canvas(plot_width=canvas_x,
+                                plot_height=canvas_y)
+        agg = cvs.raster(source=feat)
+        freq_dim = feat.dims[0]
+        freq, d, spec = agg.coords[freq_dim].values, agg.coords['datetime'].values, agg.data
+        dates = num2date(d, units='hours since 1970-01-01 00:00:00.0', calendar='gregorian')
+        return freq,dates,spec
+
+
+# #pydanticmodel output: Json file
+# class Feature(BaseModel):
+#     name: list
+
+
+# class Channel(BaseModel):
+#     name: str
+#     features: List[Feature] = []
+
+
+# class Location(BaseModel):
+#     name: str
+#     channels: List[Channel] = [] 
+
+
+# class Station(BaseModel):
+#     name: str
+#     lat: float
+#     lon: float
+#     locations: List[Location] = []
+
+
+# class Group(BaseModel):
+#     volcano: str
+#     stations: List[Station] = []
+
+
+# def get_pydanticModel(group, station, location, channel, feature_list):
+
+#     channels_data = {"name": channel, "features": feature_list}
+#     channel_models = []
+#     channel_model = Channel(**channels_data)
+#     channel_models.append(channel_model)
+
+#     location_data = {"name": location, "channels": channel_models}
+#     location_models = []
+#     location_model = Location(**location_data)
+#     location_models.append(location_model)
+
+#     stations_data = {"name": station, "lat": "42", "lon": "171",
+#                      "locations": location_models}
+#     station_models = []
+#     station_model = Station(**stations_data)
+#     station_models.append(station_model)
+
+#     group_model = Group(group=group, stations=station_models)
+
+#     # Exporting to JSON
+#     json_data = group_model.json()
+#     return json_data
+
+
+# write a function that scans LOCKERROOMROOT for 
+# available groups, stations, locations, channels, and features
+# and returns a pydantic model
+# def get_available_features():
+#     groups = os.listdir(ROOT)
+#     group_models = []
+#     for group in groups:
+#         stations = os.listdir(os.path.join(LOCKERROOMROOT, group))
+#         station_models = []
+#         for station in stations:
+#             locations = os.listdir(os.path.join(LOCKERROOMROOT, group, station))
+#             location_models = []
+#             for location in locations:
+#                 channels = os.listdir(os.path.join(LOCKERROOMROOT, group, station, location))
+#                 channel_models = []
+#                 for channel in channels:
+#                     features = os.listdir(os.path.join(LOCKERROOMROOT, group, station, location, channel))
+#                     feature_list = []
+#                     for feature in features:
+#                         feature_list.append(feature)
+#                     channel_data = {"name": channel, "features": feature_list}
+#                     channel_model = Channel(**channel_data)
+#                     channel_models.append(channel_model)
+#                 location_data = {"name": location, "channels": channel_models}
+#                 location_model = Location(**location_data)
+#                 location_models.append(location_model)
+#             station_data = {"name": station, "lat": "42", "lon": "171", "locations": location_models}
+#             station_model = Station(**station_data)
+#             station_models.append(station_model)
+#         group_data = {"volcano": group, "stations": station_models}
+#         group_model = Group(**group_data)
+#         group_models.append(group_model)
+#     return group_models
+
+# @app.get("/featureEndpoint")
+# def featureEndpoint(group: str="all", station: str="all", channel: str="all",
+#                     type: str="all"):
+#     groups = vm.get_available_volcanoes()
+
+#     station_model_list = []
+#     channel_model_list = []
+#     volcano_model_list = []
+#     for _volcano in volcanoes:
+#         streams = vm.get_available_streams(_volcano) 
+#         for _stream in streams:
+#             _, _station, _, _channel = _stream.split('.') 
+#             stream_dir = os.path.join(FEATUREDIR, _volcano, _station, _channel)
+#             try:
+#                 feature_list = os.listdir(stream_dir)
+#             except (NotADirectoryError, FileNotFoundError):
+#                 continue
+#             feature_list = sorted([str(os.path.basename(path)).split('.nc')[0] for path in feature_list])
+#             channels_data = {"name": _channel, "features":feature_list}
+#             channel_model = Channel(**channels_data)
+#             channel_model_list.append(channel_model)
+#             try:
+#                 site_info = vm.get_site_information(_station)
+#                 lat = site_info['latitude']
+#                 lon = site_info['longitude'] 
+#             except:
+#                 lat, lon = -999.9, -999.9
+#             stations_data = {"name": _station, "lat": lat, "lon": lon, "channels":channel_model_list}
+#             station_model = Station(**stations_data)
+#             station_model_list.append(station_model)
+
+#         volcano_model = Volcano(volcano=_volcano, stations=station_model_list)
+#         volcano_model_list.append(volcano_model)
+
+#     if len(volcano_model_list) == 0:
+#         return('no volcano')
+
+#     scenario_model = Scenario(scenario='VUMT', volcanoes=volcano_model_list)
+#     if volcano != "all":
+#         # return all stations for a volcano
+#         for _volcano in scenario_model.volcanoes:
+#             if _volcano.volcano == volcano:
+#                 if station == "all":
+#                     return _volcano
+#                 for _station in _volcano.stations:
+#                     if _station.name == station:
+#                         if channel == "all":
+#                             return _station
+#                         for _channel in _station.channels:
+#                             if _channel.name == channel:
+#                                 feature_list_filtered = []
+#                                 for _f in _channel.features:
+#                                     if _f in FeatureRequest.feat_dict[type]:
+#                                         feature_list_filtered.append(_f)    
+#                                 _channel.features = feature_list_filtered
+#                                 return _channel
+
+#     return scenario_model
+
+
+
+def main(argv=None):
+    parser = ArgumentParser()
+    parser.add_argument("--rootdir", default='/tmp')
+    args = parser.parse_args(argv)
+    ta = TonikAPI(args.rootdir)
+    uvicorn.run(ta.app, host="0.0.0.0", port=8003)
+
+if __name__ == "__main__":
+    main()