ents 2.3.5__tar.gz → 2.3.6__tar.gz

{ents-2.3.5 → ents-2.3.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ents
-Version: 2.3.5
+Version: 2.3.6
 Summary: Python package for Environmental NeTworked Sensor (ENTS)
 Project-URL: Homepage, https://github.com/jlab-sensing/soil-power-sensor-firmware
 Project-URL: Issues, https://github.com/jlab-sensing/soil-power-sensor-firmware/issues
@@ -12,7 +12,7 @@ Classifier: Programming Language :: Python :: 3
 Requires-Python: >=3.11
 Requires-Dist: matplotlib
 Requires-Dist: pandas
-Requires-Dist: protobuf==6.33.2
+Requires-Dist: protobuf==6.33.4
 Requires-Dist: pyserial
 Requires-Dist: requests
 Requires-Dist: scikit-learn
@@ -34,14 +34,14 @@ The soil power sensor protobuf protocol is implemented as a Python package that
 Use the following to install the `ents` package with gui via `pip`:
 
 ```bash
-pip install ents[gui]
+pip install ents
 ```
 
 You can also install the package from source with the following:
 
 ```bash
 # install package
-pip install .[gui]
+pip install .
 ```
 
 If you are planning to develop the package we recommend you install the package
@@ -51,114 +51,68 @@ reinstall it.
 
 ```bash
 # install development dependencies
-pip install -e .[gui,dev]
+pip install -e .[dev]
 ```
 
-## Usage
+To install the *deprecated* user config gui, use the following: =
+```bash
+pip install -e ents[gui]
+```
 
-The following example code demonstrates decoding the measurement message and encoding a response.
 
-```python
-from ents import encode, decode
 
-# get data encoded by the soil power sensor
-data = ...
+## Simulator (New)
 
-meas_dict = decode(data)
+The webserver `tools/http_decoder.py` can be used to decode uploaded measurements.
+
+### CLI Usage
+
+```
+usage: ents sim_generic [-h] [-v] [--url URL] --sensor SENSOR [SENSOR ...] [--min MIN] [--max MAX] --cell CELL --logger LOGGER [--start START] [--end END] [--freq FREQ] {batch,stream}
+
+positional arguments:
+  {batch,stream}        Upload mode
+
+options:
+  -h, --help            show this help message and exit
+  -v, --verbose         Print addiitional request information.
+  --url URL             URL of the dirtviz instance (default: http://localhost:8000)
+  --sensor SENSOR [SENSOR ...]
+                        Type of sensor to simulate
+  --min MIN             Minimum sensor value (default: -1.0)
+  --max MAX             Maximum sensor value (default: 1.0)
+  --cell CELL           Cell Id
+  --logger LOGGER       Logger Id
+
+Batch:
+  --start START         Start date
+  --end END             End date
+
+Stream:
+  --freq FREQ           Frequency of uploads (default: 10s)
+```
+
+### Examples
+
+You can find the available sensors in the `sensors.proto` file.
+
+Example uploading single measurement
+```
+ents sim_generic stream --sensor POWER_VOLTAGE --min 20 --max 30 --cell 1 --logger 1
+```
+
+Example uploading multiple measuremnets
+```
+ents sim_generic stream --sensor TEROS12_VWC_ADJ TEROS12_TEMP TEROS12_EC --min 10 --max 100 --cell 1 --logger 1
+```
+
+Example batch uploads
+```
+ents sim_generic batch --sensor POWER_CURRENT --cell 1 --logger 1 --start 2026-01-19 --end 2026-01-20 --freq 60
+```
 
-# process data
-...
 
-# send response
-resp_str = encode(success=True)
-```
-
-The formatting of the dictionary depends on the type of measurement sent. The key `type` is included on all measurement types and can be used to determine the type of message. See the source `*.proto` files to get the full list of types to get the full list of types and keys. A list is provided in [Message Types](#message-types). The Python protobuf API uses camel case when naming keys. The key `ts` is in ISO 8601 format as a string.
-
-## Message Types
-
-Type `power`
-```python
-meas_dict = {
-  "type": "power",
-  "loggerId": ...,
-  "cellId": ...,
-  "ts": ...,
-  "data": {
-    "voltage": ...,
-    "current": ...
-  },
-  "data_type": {
-    "voltage": float,
-    "voltage": float
-  }
-}
-```
-
-Type `teros12`
-```python
-meas_dict = {
-  "type": "teros12",
-  "loggerId": ...,
-  "cellId": ...,
-  "ts": ...,
-  "data": {
-    "vwcRaw": ...,
-    "vwcAdj": ...,
-    "temp": ...,
-    "ec": ...
-  },
-  "data_type": {
-    "vwcRaw": float,
-    "vwcAdj": float,
-    "temp": float,
-    "ec": int
-  }
-}
-```
-
-Type `bme280` with `raw=True` (default)
-```python
-meas_dict = {
-  "type": "bme280",
-  "loggerId": ...,
-  "cellId": ...,
-  "ts": ...,
-  "data": {
-    "pressure": ...,
-    "temperature": ...,
-    "humidity": ...,
-  },
-  "data_type": {
-    "pressure": int,
-    "temperature": int,
-    "humidity": int, 
-  }
-}
-```
-
-Type `bme280` with `raw=False`
-```python
-meas_dict = {
-  "type": "bme280",
-  "loggerId": ...,
-  "cellId": ...,
-  "ts": ...,
-  "data": {
-    "pressure": ...,
-    "temperature": ...,
-    "humidity": ...,
-  },
-  "data_type": {
-    "pressure": float,
-    "temperature": float,
-    "humidity": float, 
-  }
-}
-```
-
-
-## Simulator
+## Simulator (Old)
 
 Simulate WiFi sensor uploads without requiring ENTS hardware.
 

{ents-2.3.5 → ents-2.3.6}/README.md

@@ -8,14 +8,14 @@ The soil power sensor protobuf protocol is implemented as a Python package that
 Use the following to install the `ents` package with gui via `pip`:
 
 ```bash
-pip install ents[gui]
+pip install ents
 ```
 
 You can also install the package from source with the following:
 
 ```bash
 # install package
-pip install .[gui]
+pip install .
 ```
 
 If you are planning to develop the package we recommend you install the package
@@ -25,114 +25,68 @@ reinstall it.
 
 ```bash
 # install development dependencies
-pip install -e .[gui,dev]
+pip install -e .[dev]
 ```
 
-## Usage
+To install the *deprecated* user config gui, use the following: =
+```bash
+pip install -e ents[gui]
+```
 
-The following example code demonstrates decoding the measurement message and encoding a response.
 
-```python
-from ents import encode, decode
 
-# get data encoded by the soil power sensor
-data = ...
+## Simulator (New)
 
-meas_dict = decode(data)
+The webserver `tools/http_decoder.py` can be used to decode uploaded measurements.
+
+### CLI Usage
+
+```
+usage: ents sim_generic [-h] [-v] [--url URL] --sensor SENSOR [SENSOR ...] [--min MIN] [--max MAX] --cell CELL --logger LOGGER [--start START] [--end END] [--freq FREQ] {batch,stream}
+
+positional arguments:
+  {batch,stream}        Upload mode
+
+options:
+  -h, --help            show this help message and exit
+  -v, --verbose         Print addiitional request information.
+  --url URL             URL of the dirtviz instance (default: http://localhost:8000)
+  --sensor SENSOR [SENSOR ...]
+                        Type of sensor to simulate
+  --min MIN             Minimum sensor value (default: -1.0)
+  --max MAX             Maximum sensor value (default: 1.0)
+  --cell CELL           Cell Id
+  --logger LOGGER       Logger Id
+
+Batch:
+  --start START         Start date
+  --end END             End date
+
+Stream:
+  --freq FREQ           Frequency of uploads (default: 10s)
+```
+
+### Examples
+
+You can find the available sensors in the `sensors.proto` file.
+
+Example uploading single measurement
+```
+ents sim_generic stream --sensor POWER_VOLTAGE --min 20 --max 30 --cell 1 --logger 1
+```
+
+Example uploading multiple measuremnets
+```
+ents sim_generic stream --sensor TEROS12_VWC_ADJ TEROS12_TEMP TEROS12_EC --min 10 --max 100 --cell 1 --logger 1
+```
+
+Example batch uploads
+```
+ents sim_generic batch --sensor POWER_CURRENT --cell 1 --logger 1 --start 2026-01-19 --end 2026-01-20 --freq 60
+```
 
-# process data
-...
 
-# send response
-resp_str = encode(success=True)
-```
-
-The formatting of the dictionary depends on the type of measurement sent. The key `type` is included on all measurement types and can be used to determine the type of message. See the source `*.proto` files to get the full list of types to get the full list of types and keys. A list is provided in [Message Types](#message-types). The Python protobuf API uses camel case when naming keys. The key `ts` is in ISO 8601 format as a string.
-
-## Message Types
-
-Type `power`
-```python
-meas_dict = {
-  "type": "power",
-  "loggerId": ...,
-  "cellId": ...,
-  "ts": ...,
-  "data": {
-    "voltage": ...,
-    "current": ...
-  },
-  "data_type": {
-    "voltage": float,
-    "voltage": float
-  }
-}
-```
-
-Type `teros12`
-```python
-meas_dict = {
-  "type": "teros12",
-  "loggerId": ...,
-  "cellId": ...,
-  "ts": ...,
-  "data": {
-    "vwcRaw": ...,
-    "vwcAdj": ...,
-    "temp": ...,
-    "ec": ...
-  },
-  "data_type": {
-    "vwcRaw": float,
-    "vwcAdj": float,
-    "temp": float,
-    "ec": int
-  }
-}
-```
-
-Type `bme280` with `raw=True` (default)
-```python
-meas_dict = {
-  "type": "bme280",
-  "loggerId": ...,
-  "cellId": ...,
-  "ts": ...,
-  "data": {
-    "pressure": ...,
-    "temperature": ...,
-    "humidity": ...,
-  },
-  "data_type": {
-    "pressure": int,
-    "temperature": int,
-    "humidity": int, 
-  }
-}
-```
-
-Type `bme280` with `raw=False`
-```python
-meas_dict = {
-  "type": "bme280",
-  "loggerId": ...,
-  "cellId": ...,
-  "ts": ...,
-  "data": {
-    "pressure": ...,
-    "temperature": ...,
-    "humidity": ...,
-  },
-  "data_type": {
-    "pressure": float,
-    "temperature": float,
-    "humidity": float, 
-  }
-}
-```
-
-
-## Simulator
+## Simulator (Old)
 
 Simulate WiFi sensor uploads without requiring ENTS hardware.
 

{ents-2.3.5 → ents-2.3.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ents"
-version = "2.3.5"
+version = "2.3.6"
 authors = [
   { name="John Madden", email="jmadden173@pm.me" },
 ]
@@ -17,7 +17,7 @@ classifiers = [
   "Operating System :: OS Independent",
 ]
 dependencies = [
-  'protobuf==6.33.2',
+  'protobuf==6.33.4',
   'matplotlib',
   'pandas',
   'pyserial',

{ents-2.3.5 → ents-2.3.6}/src/ents/calibrate/recorder.py

@@ -16,7 +16,7 @@ import socket
 import serial
 from typing import Tuple
 from tqdm import tqdm
-from ..proto import decode_measurement
+from ..proto.sensor import decode_repeated_sensor_measurements
 
 
 class SerialController:
@@ -116,10 +116,10 @@ class SoilPowerSensorController(SerialController):
 
         reply = self.ser.read(resp_len)  # read said measurment
 
-        meas_dict = decode_measurement(reply)  # decode using protobuf
+        meas_dict = decode_repeated_sensor_measurements(reply)  # decode using protobuf
 
-        voltage_value = meas_dict["data"]["voltage"]
-        current_value = meas_dict["data"]["current"]
+        voltage_value = meas_dict["measurements"][0]["decimal"]
+        current_value = meas_dict["measurements"][1]["decimal"]
 
         return float(voltage_value), float(current_value)
 

{ents-2.3.5 → ents-2.3.6}/src/ents/cli.py

@@ -36,7 +36,7 @@ from .proto.sensor import (
     decode_sensor_response,
 )
 
-from .simulator.node import NodeSimulator
+from .simulator.node import NodeSimulator, NodeSimulatorGeneric
 
 
 def entry():
@@ -68,15 +68,22 @@ def create_sim_generic_parser(subparsers):
     """
 
     sim_p = subparsers.add_parser("sim_generic", help="Simluate generic sensor uploads")
+
+    sim_p.add_argument(
+        "-v",
+        "--verbose",
+        action="store_true",
+        help="Print addiitional request information.",
+    )
+
     sim_p.add_argument(
         "--url",
-        required=True,
+        default="http://localhost:8000/api/sensor",
         type=str,
         help="URL of the dirtviz instance (default: http://localhost:8000)",
     )
     sim_p.add_argument(
-        "--mode",
-        required=True,
+        "mode",
         choices=["batch", "stream"],
         type=str,
         help="Upload mode",
@@ -88,6 +95,7 @@ def create_sim_generic_parser(subparsers):
         nargs="+",
         help="Type of sensor to simulate",
     )
+
     sim_p.add_argument(
         "--min", type=float, default=-1.0, help="Minimum sensor value (default: -1.0)"
     )
@@ -96,11 +104,16 @@ def create_sim_generic_parser(subparsers):
     )
     sim_p.add_argument("--cell", required=True, type=int, help="Cell Id")
     sim_p.add_argument("--logger", required=True, type=int, help="Logger Id")
-    sim_p.add_argument("--start", type=str, help="Start date")
-    sim_p.add_argument("--end", type=str, help="End date")
-    sim_p.add_argument(
+
+    batch = sim_p.add_argument_group("Batch")
+    batch.add_argument("--start", type=str, help="Start date")
+    batch.add_argument("--end", type=str, help="End date")
+
+    stream = sim_p.add_argument_group("Stream")
+    stream.add_argument(
         "--freq", default=10.0, type=float, help="Frequency of uploads (default: 10s)"
     )
+
     sim_p.set_defaults(func=simulate_generic)
 
     return sim_p
@@ -149,7 +162,12 @@ def create_sim_parser(subparsers):
 
 
 def simulate_generic(args):
-    simulation = NodeSimulator(
+    if args.verbose:
+        print("Arguments:")
+        print(args)
+        print()
+
+    simulation = NodeSimulatorGeneric(
         cell=args.cell,
         logger=args.logger,
         sensors=args.sensor,
@@ -184,8 +202,17 @@ def simulate_generic(args):
                 dt = datetime.now()
                 ts = int(dt.timestamp())
                 simulation.measure(ts)
+
                 while simulation.send_next(args.url):
                     print(simulation)
+
+                if args.verbose:
+                    print("Request")
+                    print(simulation.last_request())
+                    print("\nResponse")
+                    print(simulation.last_response())
+                    print()
+
                 time.sleep(args.freq)
         except KeyboardInterrupt as _:
             print("Stopping simulation")

ents-2.3.6/src/ents/dirtviz/__init__.py

@@ -0,0 +1,8 @@
+from .client import BackendClient
+
+from .plots import plot_data
+
+__all__ = [
+    "BackendClient",
+    "plot_data",
+]

ents-2.3.6/src/ents/dirtviz/client.py

@@ -0,0 +1,223 @@
+"""Client interface with dirtviz.
+
+TODO:
+- Add caching of data
+"""
+
+from datetime import datetime
+
+import pandas as pd
+
+import requests
+
+
+class Cell:
+    """Class representing a cell in the Dirtviz API."""
+
+    def __init__(self, data: str):
+        """Initialize the Cell object from a cell ID.
+
+        Args:
+            data: json data from the Dirtviz API containing cell information.
+        """
+
+        self.id = data["id"]
+        self.name = data["name"]
+        self.location = data["location"]
+        self.latitude = data["latitude"]
+        self.longitude = data["longitude"]
+
+    def __repr__(self):
+        return f"Cell(id={self.cell_id}, name={self.name})"
+
+
+class BackendClient:
+    """Client for interacting with the Dirtviz API."""
+
+    def __init__(self, base_url: str = "https://dirtviz.jlab.ucsc.edu/api/"):
+        """Initialize the BackendClient.
+
+        Sets the base URL for the API. Defaults to the Dirtviz API.
+        """
+
+        self.base_url = base_url
+
+    def get(self, endpoint: str, params: dict = None) -> dict:
+        """Get request to the API.
+
+        Args:
+            endpoint: The API endpoint to request.
+            params: Optional parameters for the request.
+
+        Returns:
+            A dictionary containing the response data.
+        """
+
+        url = f"{self.base_url}{endpoint}"
+        response = requests.get(url, params=params)
+        response.raise_for_status()
+
+        return response.json()
+
+    @staticmethod
+    def time_to_params(start: datetime, end: datetime) -> dict:
+        """Puts start and end datetime into an API paramter dictionary
+
+        Args:
+            dt: The datetime object to format.
+
+        Returns:
+            A string representing the formatted datetime.
+        """
+
+        timestamp_format = "%a, %d %b %Y %H:%M:%S GMT"
+
+        start_str = start.strftime(timestamp_format)
+        end_str = end.strftime(timestamp_format)
+
+        params = {
+            "startTime": start_str,
+            "endTime": end_str,
+        }
+
+        return params
+
+    def power_data(self, cell: Cell, start: datetime, end: datetime) -> pd.DataFrame:
+        """Gets power data for a specific cell by name.
+
+        Args:
+            cell: The Cell object for which to get power data.
+            start: The start date of the data.
+            end: The end date of the data.
+
+        Returns:
+            A pandas DataFrame containing the power data.
+        """
+
+        endpoint = f"/power/{cell.id}"
+
+        params = self.time_to_params(start, end)
+
+        data = self.get(endpoint, params=params)
+
+        data_df = pd.DataFrame(data)
+        data_df["timestamp"] = pd.to_datetime(data_df["timestamp"])
+
+        return data_df
+
+    def teros_data(self, cell: Cell, start: datetime, end: datetime) -> pd.DataFrame:
+        """Gets teros data for a specific cell
+
+        Args:
+            cell: The Cell object for which to get teros data.
+            start: The start date of the data.
+            end: The end date of the data.
+
+        Returns:
+            A pandas DataFrame containing the teros data with columns vwc_raw,
+            vwc_adj, temp, ec.
+        """
+
+        endpoint = f"/teros/{cell.id}"
+
+        params = self.time_to_params(start, end)
+
+        data = self.get(endpoint, params=params)
+
+        data_df = pd.DataFrame(data)
+        data_df["timestamp"] = pd.to_datetime(data_df["timestamp"])
+
+        return data_df
+
+    def sensor_data(
+        self,
+        cell: Cell,
+        name: str,
+        meas: str,
+        start: datetime,
+        end: datetime,
+        resample: str = "none",
+    ) -> pd.DataFrame:
+        """Gets generic sensor data for a specific cell
+
+        Args:
+            cell: The Cell object for which to get sensor data.
+            name: Name of the sensor (e.g., "power", "teros").
+            meas: The measurement type (e.g., "v", "i", "vwc", "temp", "ec").
+            start: The start date of the data.
+            end: The end date of the data.
+
+        Returns:
+            A pandas DataFrame containing the sensor data.
+        """
+
+        endpoint = "/sensor/"
+
+        params = {
+            "cellId": cell.id,
+            "name": name,
+            "measurement": meas,
+        }
+
+        params = params | self.time_to_params(start, end)
+
+        data = self.get(endpoint, params=params)
+
+        data_df = pd.DataFrame(data)
+        data_df["timestamp"] = pd.to_datetime(data_df["timestamp"])
+
+        return data_df
+
+    def cell_from_id(self, cell_id: int) -> Cell | None:
+        """Get a Cell object from its ID.
+
+        Args:
+            cell_id: The ID of the cell.
+
+        Returns:
+            A Cell object. None if the cell does not exist.
+        """
+
+        cell_list = self.cells()
+
+        for cell in cell_list:
+            if cell.id == cell_id:
+                return cell
+
+        return None
+
+    def cell_from_name(self, name: str) -> Cell | None:
+        """Get a Cell object from its name.
+
+        Args:
+            name: The name of the cell.
+
+        Returns:
+            A Cell object. None if the cell does not exist.
+        """
+
+        cell_list = self.cells()
+
+        for cell in cell_list:
+            if cell.name == name:
+                return cell
+
+        return None
+
+    def cells(self) -> list[Cell]:
+        """Gets a list of all cells from the API.
+
+        Returns:
+            A list of Cell objects.
+        """
+
+        cell_list = []
+
+        endpoint = "/cell/id"
+        cell_data_list = self.get(endpoint)
+
+        for c in cell_data_list:
+            cell = Cell(c)
+            cell_list.append(cell)
+
+        return cell_list

ents-2.3.6/src/ents/dirtviz/plots.py

@@ -0,0 +1,50 @@
+"""Common plots for data on dirtviz.
+
+Still need to fill in with common functions:
+- Plot power (voltage, current, calcualted power)
+- Plot teros12
+- Plot bme280
+- Plot all (take all available sensors)
+- Plot group (find mean/stddev for each point)
+
+Can pull from personal scripts from SenSys (jtmadden)
+"""
+
+import pandas as pd
+
+import matplotlib.pyplot as plt
+
+
+# matplotlib formatting
+# plt.rcParams["font.size"] = 7
+# plt.rcParams['font.weight'] = 'medium'
+# plt.ion()
+
+
+def plot_data(data: list[pd.DataFrame], name, **kwargs):
+    """Plots data from one or many cells
+
+    Args:
+        data: List of dataframes
+        name: Column or measurement name
+    """
+
+    fig, ax = plt.subplots()
+    for d in data:
+        ax.plot(d["timestamp"], d[name], **kwargs)
+
+    ax.axhline(
+        y=0,
+        color="black",
+        linewidth=2,
+        dashes=(2, 2),
+    )
+
+    ax.set_xlabel("Timestamp")
+    ax.set_ylabel(name)
+
+    ax.grid()
+
+    plt.show(block=False)
+
+    return ax

{ents-2.3.5 → ents-2.3.6}/src/ents/simulator/node.py

@@ -172,6 +172,8 @@ class NodeSimulatorGeneric:
     measurements: list[bytes] = []
     # all responses
     responses: list[str] = []
+    # all requests in format (headers, body)
+    requests: list[tuple[str, str]] = []
 
     # metrics for uploads
     metrics: dict[str, int] = {
@@ -233,14 +235,18 @@ class NodeSimulatorGeneric:
 
         # get next measurement
         try:
-            meas = self.measurements.pop()
+            meas = self.measurement_buffer.pop()
         except IndexError as _:
             return False
 
-        headers = {"Content-Type": "application/octet-stream"}
+        headers = {
+            "Content-Type": "application/octet-stream",
+            "SensorVersion": "2",
+        }
         result = requests.post(url, data=meas, headers=headers)
 
         # store result
+        self.requests.append((result.request.headers, result.request.body))
         self.responses.append(result.text)
         self.metrics["total_requests"] += 1
         if result.status_code == 200:
@@ -279,4 +285,36 @@ class NodeSimulatorGeneric:
             )
 
         serialized = encode_repeated_sensor_measurements(meas)
+        self.measurements.append(serialized)
         self.measurement_buffer.append(serialized)
+
+    def last_measurement(self) -> bytes:
+        """Gets the last encoded measurement.
+
+
+        Returns:
+            Last encoded measurement.
+        """
+
+        return self.measurements[-1]
+
+    def last_request(self) -> str:
+        """Gets the last sent request.
+
+        Returns:
+            Formatted headers and body.
+        """
+
+        headers = self.requests[-1][0]
+        body = self.requests[-1][1]
+        request_str = f"{headers}\n\n{body}"
+        return request_str
+
+    def last_response(self) -> str:
+        """Gets the last response.
+
+        Returns:
+            Response from server.
+        """
+
+        return self.responses[-1]