marpledata 3.2.2__tar.gz → 3.2.4__tar.gz

marpledata-3.2.4/.flake8

@@ -0,0 +1,5 @@
+[flake8]
+max-line-length = 1000
+extend-ignore = E203
+exclude =
+    .venv

{marpledata-3.2.2 → marpledata-3.2.4}/AGENTS.md

@@ -17,6 +17,9 @@ This directory contains the Python SDK package published as `marpledata`.
 - Run tests with output: `uv run pytest -vs`
 - Build docs: `uv run --group docs sphinx-build -b html docs docs/_build/html`
 - Build package: `uv build`
+- Fix formatting: `uv run isort src tests && uv run black src tests`
+- Verify lint/format/types (mirror CI):
+  `uv run isort src tests --check --diff && uv run flake8 --config .flake8 src tests && uv run black --check src tests && uv run mypy --install-types --non-interactive`
 
 Run commands from `python/` unless a command explicitly says otherwise.
 

{marpledata-3.2.2 → marpledata-3.2.4}/CHANGELOG.md

@@ -5,6 +5,24 @@ All notable changes to the Python SDK package `marpledata` will be documented in
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.2.4] - 2026-06-19
+
+### Added
+
+- Added `DataStream.add_dataset`, `Dataset.upsert_signals`, `Dataset.append`, and `Dataset.cool` for realtime ingest.
+- Extended `Dataset.wait_for_import` to recognize `COOLING` as a busy status.
+
+### Changed
+
+- `DB.add_dataset`, `DB.upsert_signals`, and `DB.dataset_append` now delegate to the DataStream and Dataset methods.
+- Fixed broken `DB.download_signal` compatibility path (`get_parquet_files` → `list_parquet_files`).
+
+## [3.2.3] - 2026-06-08
+
+### Changed
+
+- Bugfix build artifact `insight.py`
+
 ## [3.2.2] - 2026-06-01
 
 ### Changed

{marpledata-3.2.2 → marpledata-3.2.4}/CONTRIBUTING.md

@@ -1,5 +1,25 @@
 ## Development (Python)
 
+### Formatting, linting, and typing
+
+Checks run on `src/` and `tests/` in GitLab CI (`python:lint`, `python:typing`).
+
+Fix formatting locally:
+
+```bash
+uv run isort src tests
+uv run black src tests
+```
+
+Verify (mirror CI):
+
+```bash
+uv run isort src tests --check --diff
+uv run flake8 --config .flake8 src tests
+uv run black --check src tests
+uv run mypy --install-types --non-interactive
+```
+
 ### Testing
 
 The testing suite runs against a real linked DB & Insight deployment on our SaaS (e.g. Castro Comrades).
@@ -44,33 +64,25 @@ Only once on your laptop
   - `username: __token__`
   - `password: pypi-XXXXXXXXXXXXXXXXXXXXXXXXXXXX` (see 1Password)
 
-For every build
+### Publishing Production
 
-- `uv version x.y.z.devi`
+- Make sure you don't have any local changes
+- Delete `./dist` folder
+- `uv version x.y.z(.devi)`
 - bump version in `__version__` variable
 - `uv build`
-- `uv publish --index testpypi`
+- `uv publish` ⚠ **Impacts users, be careful**
+  - `username: __token__`
+  - `password: pypi-XXXXXXXXXXXXXXXXXXXXXXXXXXXX` (see 1Password)
+- Run the GitLab pipeline `pages` to build & release the docs
 
 **Versioning**
 
 To ensure pip correctly updates our package, correct versioning is important. Use `major.minor.patch`
 
-- `major` when they make incompatible API changes
-- `minor` when they add functionality in a backwards-compatible manner
-- `patch` when they make backwards-compatible bug fixes
-
 **Recommended test workflow**
 
 - Publish to test Pypi
 - Open a docker container with the desired python version: `docker run -it python:3.11-alpine sh`
 - Install the test pypi package `pip install --upgrade -i --pre https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple marpledata`
 - If you want to publish again, change the version to one that has not been used before. Even if you delete a build via [the UI](https://test.pypi.org/manage/project/marpledata/releases/) you cannot publish that version again. For testing, you could use something like `x.y.z.dev1`, `x.y.z.dev2`, `x.y.z.dev3`, ...
-
-### Publishing (real Pypi)
-
-⚠ **Impacts users, be careful**
-
-- check version: `pyproject.toml:version`, `__init__.py:__version__`
-- `uv build`
-- `uv publish --token pypi-XXXXXXXXXXXXXXXXXXXXXXXXXXXX` (see 1Password)
-- Run the GitLab pipeline `pages` to build & release the docs

{marpledata-3.2.2 → marpledata-3.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: marpledata
-Version: 3.2.2
+Version: 3.2.4
 Summary: Marple SDK for Python
 Project-URL: Homepage, https://www.marpledata.com/
 Project-URL: Documentation, https://marpledata.gitlab.io/marple-sdk/
@@ -181,9 +181,10 @@ if len(datasets) > 0:
 
 For live/realtime streams (creating and appending data):
 
-- **Create an empty dataset**: `db.add_dataset(stream_key, dataset_name, metadata=None)`
-- **Upsert signal definitions**: `db.upsert_signals(stream_key, dataset_id, signals=[...])`
-- **Append timeseries data**: `db.dataset_append(stream_key, dataset_id, data=df, shape="long"|"wide"|None)`
+- **Create an empty dataset**: `stream.add_dataset(dataset_name, metadata=None)`
+- **Upsert signal definitions**: `dataset.upsert_signals(signals=[...])`
+- **Append timeseries data**: `dataset.append(data=df, shape="long"|"wide"|None)`
+- **Cool a realtime dataset**: `dataset.cool()` then `dataset.wait_for_import()` to wait until `FINISHED`
 
 ### Calling endpoints directly
 

{marpledata-3.2.2 → marpledata-3.2.4}/README.md

@@ -147,9 +147,10 @@ if len(datasets) > 0:
 
 For live/realtime streams (creating and appending data):
 
-- **Create an empty dataset**: `db.add_dataset(stream_key, dataset_name, metadata=None)`
-- **Upsert signal definitions**: `db.upsert_signals(stream_key, dataset_id, signals=[...])`
-- **Append timeseries data**: `db.dataset_append(stream_key, dataset_id, data=df, shape="long"|"wide"|None)`
+- **Create an empty dataset**: `stream.add_dataset(dataset_name, metadata=None)`
+- **Upsert signal definitions**: `dataset.upsert_signals(signals=[...])`
+- **Append timeseries data**: `dataset.append(data=df, shape="long"|"wide"|None)`
+- **Cool a realtime dataset**: `dataset.cool()` then `dataset.wait_for_import()` to wait until `FINISHED`
 
 ### Calling endpoints directly
 

{marpledata-3.2.2 → marpledata-3.2.4}/docs/api/marple.db.Signal.rst

@@ -13,9 +13,10 @@
 
    .. autosummary::
    
-      ~Signal.cache_parquet
+      ~Signal.download
+      ~Signal.from_dict
       ~Signal.get_data
-      ~Signal.list_parquet_files
+      ~Signal.get_parquet_files
    
    
 

{marpledata-3.2.2 → marpledata-3.2.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "marpledata"
-version = "3.2.2"
+version = "3.2.4"
 description = "Marple SDK for Python"
 authors = [
   { name = "Matthias Baert", email = "support@marpledata.com" },
@@ -56,12 +56,33 @@ explicit = true
 
 [dependency-groups]
 dev = [
+    "black>=24.2.0",
+    "flake8>=7.0.0",
     "h5py>=3.15.1",
+    "isort>=5.13.2",
+    "mypy>=1.8.0",
     "pytest>=9.0.2",
     "pytest-cov>=7.0.0",
+    "types-requests>=2.32.0",
 ]
 docs = [
   "autodoc-pydantic>=2.2.0",
   "pydata-sphinx-theme>=0.15.4",
   "sphinx>=7.2.6",
 ]
+
+[tool.black]
+line-length = 113
+target-version = ["py311"]
+
+[tool.isort]
+profile = "black"
+src_paths = ["src", "tests"]
+
+[tool.mypy]
+mypy_path = "src"
+files = ["src/marple", "tests"]
+ignore_missing_imports = true
+allow_untyped_globals = true
+explicit_package_bases = true
+

{marpledata-3.2.2 → marpledata-3.2.4}/src/marple/__init__.py

@@ -3,4 +3,4 @@ from .db import DB
 from .insight import Insight
 
 __all__ = ["DB", "Insight", "db"]
-__version__ = "3.2.2"
+__version__ = "3.2.4"

{marpledata-3.2.2 → marpledata-3.2.4}/src/marple/db/__init__.py

@@ -1,29 +1,18 @@
-import logging
 import warnings
 from functools import wraps
-from io import BytesIO
 from pathlib import Path
 from typing import Literal, Optional
 
-import numpy as np
 import pandas as pd
-import pyarrow as pa
-import pyarrow.parquet as pq
-from marple.db.constants import (
-    COL_SIG,
-    COL_TIME,
-    COL_VAL,
-    COL_VAL_TEXT,
-    SAAS_URL,
-    SCHEMA,
-)
+from pydantic import ValidationError
+from requests import Response
+from requests.exceptions import ConnectionError
+
+from marple.db.constants import SAAS_URL, SCHEMA
 from marple.db.dataset import Dataset, DatasetList
 from marple.db.datastream import DataStream
 from marple.db.signal import Signal
 from marple.utils import DBClient, validate_response
-from pydantic import ValidationError
-from requests import Response
-from requests.exceptions import ConnectionError
 
 __all__ = ["DB", "DataStream", "Dataset", "DatasetList", "Signal", "SCHEMA"]
 
@@ -191,7 +180,11 @@ class DB:
         if isinstance(stream_key, int):
             return self._streams.get(stream_key)
         return next(
-            (s for s in self._streams.values() if s.name.lower() == stream_key.lower() or str(s.id) == stream_key),
+            (
+                s
+                for s in self._streams.values()
+                if s.name.lower() == stream_key.lower() or str(s.id) == stream_key
+            ),
             None,
         )
 
@@ -229,8 +222,8 @@ class DB:
         if stream_key is not None:
             return self.get_stream(stream_key).get_datasets()
         r = self.get(f"/datapool/{self.client.datapool}/datasets")
-        r = validate_response(r, f"Failed to get datasets for datapool {self.client.datapool}")
-        return DatasetList.from_dicts(self.client, r)
+        datasets = validate_response(r, f"Failed to get datasets for datapool {self.client.datapool}")
+        return DatasetList.from_dicts(self.client, datasets)
 
     def get_dataset(self, dataset_id: int | None = None, dataset_path: str | None = None) -> Dataset:
         """
@@ -331,7 +324,7 @@ class DB:
         )
         if signal is None:
             raise Exception("Signal not found")
-        return signal.get_parquet_files(refresh_cache)
+        return signal.list_parquet_files(refresh_cache)
 
     def delete_dataset(self, dataset_id: int | None, dataset_path: str | None):
         """
@@ -373,17 +366,11 @@ class DB:
         Returns the ID of the newly created dataset.
 
         Use `dataset_append` to add data to the dataset and `upsert_signals` to define signals.
+        Use `cool` to finalize the dataset when done.
 
         To add datasets from a file to a file stream, use `push_file` instead.
         """
-        stream_id = self._get_stream_id(stream_key)
-        r = self.post(
-            f"/stream/{stream_id}/dataset/add",
-            json={"dataset_name": dataset_name, "metadata": metadata or {}},
-        )
-        r_json = validate_response(r, "Add dataset failed")
-
-        return r_json["dataset_id"]
+        return self.get_stream(stream_key).add_dataset(dataset_name, metadata).id
 
     def upsert_signals(self, stream_key: str | int, dataset_id: int, signals: list[dict]) -> None:
         """
@@ -395,10 +382,7 @@ class DB:
         - `description`: (optional) Description of the signal
         - `[any metadata key]`: (optional) Any metadata value
         """
-        stream_id = self._get_stream_id(stream_key)
-
-        r = self.post(f"/stream/{stream_id}/dataset/{dataset_id}/signals", json=signals)
-        validate_response(r, "Upsert signals failed")
+        self.get_dataset(dataset_id=dataset_id).upsert_signals(signals)
 
     def dataset_append(
         self,
@@ -423,62 +407,19 @@ class DB:
         - `"wide"` format: Each row represents a single time point with multiple signals as columns. Expects at least a `time` column.
 
         """
-        stream_id = self._get_stream_id(stream_key)
+        self.get_dataset(dataset_id=dataset_id).append(data, shape)
 
-        if self._detect_shape(shape, data) == "wide":
-            if COL_TIME not in data.columns:
-                raise ValueError("DataFrame must contain a time column")
-            table = _wide_to_long(data)
-        else:
-            if COL_TIME not in data.columns or COL_SIG not in data.columns:
-                raise Exception(f"DataFrame must contain {COL_TIME} and {COL_SIG} columns")
-            if not (COL_VAL in data.columns or COL_VAL_TEXT in data.columns):
-                raise Exception(f"DataFrame must contain at least one of {COL_VAL} or {COL_VAL_TEXT} columns")
-            value = pd.to_numeric(data[COL_VAL], errors="coerce") if COL_VAL in data.columns else pa.nulls(len(data))
-            value_text = data[COL_VAL_TEXT] if COL_VAL_TEXT in data.columns else pa.nulls(len(data))
-            table = pa.Table.from_arrays([data[COL_TIME], data[COL_SIG], value, value_text], schema=SCHEMA)
-
-        parquet_buffer = BytesIO()
-        pq.write_table(table, parquet_buffer)
-        parquet_buffer.seek(0)
-
-        # Send as multipart/form-data
-        files = {"file": ("data.parquet", parquet_buffer, "application/octet-stream")}
-
-        r = self.post(f"/stream/{stream_id}/dataset/{dataset_id}/append", files=files)
-        validate_response(r, "Append data failed")
-
-    # Internal functions #
-
-    @staticmethod
-    def _detect_shape(shape: Optional[Literal["long", "wide"]], df: pd.DataFrame) -> Literal["long", "wide"]:
-        if shape is not None:
-            return shape
-
-        if "signal" in df.columns and ((COL_VAL in df.columns) or (COL_VAL_TEXT in df.columns)):
-            return "long"
-        else:
-            return "wide"
-
-
-def _wide_to_long(df: pd.DataFrame) -> pa.Table:
-    signals = []
-    time = pa.array(df[COL_TIME], type=pa.int64())
-    for col in df.columns:
-        if col == COL_TIME:
-            continue
-        if (value := _to_numeric(df[col])) is not None:
-            (value, value_text) = (value.to_numpy().astype(np.float64), pa.nulls(len(time)))
-        else:
-            (value, value_text) = (pa.nulls(len(time)), df[col].fillna("").to_numpy().astype(str))
-        signals.append(pa.Table.from_arrays([time, [col] * len(time), value, value_text], schema=SCHEMA))
-    return pa.concat_tables(signals)
-
-
-def _to_numeric(col: pd.Series) -> pd.Series | None:
-    if pd.api.types.is_numeric_dtype(col.dtype):
-        return col
-    null_count = col.isnull().sum()
-    numeric_col = pd.to_numeric(col, errors="coerce")
-    is_numeric = (numeric_col.isnull().sum() - null_count) / max(len(col), 1) < 0.2
-    return numeric_col if is_numeric else None
+    def dataset_cool(self, stream_key: str | int, dataset_id: int) -> Dataset:
+        """
+        Move all realtime data to cold storage and finalize the specified realtime dataset.
+
+        Cooling is started asynchronously on the server. Poll completion with
+        `Dataset.wait_for_import`.
+
+        Only realtime datasets in LIVE or COOLING_FAILED status can be cooled. After
+        cooling completes, the dataset no longer accepts appends and
+        `import_status` becomes FINISHED.
+
+        Returns the current dataset state (typically `import_status == "COOLING"`).
+        """
+        return self.get_dataset(dataset_id=dataset_id).cool()

{marpledata-3.2.2 → marpledata-3.2.4}/src/marple/db/dataset.py

@@ -2,17 +2,19 @@ import re
 import time
 import warnings
 from collections import UserList
+from io import BytesIO
 from pathlib import Path
-from typing import Callable, Iterable, Literal, Optional, Sequence
+from typing import Callable, Iterable, Literal, Optional
 from urllib import parse, request
 
+import numpy as np
 import pandas as pd
-from pandas._typing import AggFuncType, Frequency
-from pydantic import BaseModel, ConfigDict, Field, PrivateAttr
-from pydantic import ValidationError
+import pyarrow as pa
 import pyarrow.parquet as pq
-from marple.db.constants import COL_TIME, COL_VAL
+from pandas._typing import AggFuncType, Frequency
+from pydantic import BaseModel, ConfigDict, Field, PrivateAttr, ValidationError
 
+from marple.db.constants import COL_SIG, COL_TIME, COL_VAL, COL_VAL_TEXT, SCHEMA
 from marple.db.signal import Signal
 from marple.utils import DBClient, validate_response
 
@@ -21,6 +23,7 @@ BUSY_STATUSES = [
     "IMPORTING",
     "POST_PROCESSING",
     "UPDATING_ICEBERG",
+    "COOLING",
 ]
 
 
@@ -67,7 +70,9 @@ class Dataset(BaseModel):
         self._client = client
 
     @classmethod
-    def fetch(cls, client: DBClient, dataset_id: int | None = None, dataset_path: str | None = None) -> "Dataset":
+    def fetch(
+        cls, client: DBClient, dataset_id: int | None = None, dataset_path: str | None = None
+    ) -> "Dataset":
         """
         Fetch a dataset by its ID or path.
 
@@ -130,24 +135,24 @@ class Dataset(BaseModel):
         """
         if signal_names is None:
             return self._get_all_signals()
-        signals = self._client.find_matching_signals(signal_names)
-        if len(signals) == 0:
+        signal_ids = self._client.find_matching_signals(signal_names)
+        if len(signal_ids) == 0:
             return []
 
         r = self._client.get(
             f"/stream/{self.datastream_id}/dataset/{self.id}/signals",
-            params={"signal_ids": list(signals.values())},
+            params={"signal_ids": list(signal_ids.values())},
         )
-        signals = []
+        result: list[Signal] = []
         for response in validate_response(r, "Failed to get signals by name"):
             try:
                 signal = Signal(self._client, self.datastream_id, self.id, **response)
-                signals.append(signal)
+                result.append(signal)
             except ValidationError as e:
                 warnings.warn(f"Failed to create signal {response['name']} (id {response['id']}): {e}")
                 continue
             self._signals[signal.id] = signal
-        return signals
+        return result
 
     def get_data(
         self,
@@ -236,12 +241,90 @@ class Dataset(BaseModel):
         validate_response(r, "Update metadata failed")
         return self.fetch(self._client, self.id)
 
+    def upsert_signals(self, signals: list[dict]) -> None:
+        """
+        Add signals to this dataset or update existing ones.
+
+        Each signal in the `signals` list should be a dictionary with the following keys:
+        - `signal`: Name of the signal
+        - `unit`: (optional) Unit of the signal
+        - `description`: (optional) Description of the signal
+        - `[any metadata key]`: (optional) Any metadata value
+        """
+        r = self._client.post(f"/stream/{self.datastream_id}/dataset/{self.id}/signals", json=signals)
+        validate_response(r, "Upsert signals failed")
+
+    def append(
+        self,
+        data: pd.DataFrame,
+        shape: Optional[Literal["wide", "long"]] = None,
+    ) -> None:
+        """
+        Append new data to this realtime dataset.
+
+        `data` is a DataFrame with the following columns. It can be in either "long"
+        or "wide" format. If `shape` is not specified, the format is automatically
+        detected:
+
+        - `"long"` format: Each row represents a single measurement for a single signal
+          at a specific time. Expects `time`, `signal`, and at least one of `value` or
+          `value_text`.
+        - `"wide"` format: Each row represents a single time point with multiple signals
+          as columns. Expects at least a `time` column.
+        """
+        if _detect_shape(shape, data) == "wide":
+            if COL_TIME not in data.columns:
+                raise ValueError("DataFrame must contain a time column")
+            table = _wide_to_long(data)
+        else:
+            if COL_TIME not in data.columns or COL_SIG not in data.columns:
+                raise ValueError(f"DataFrame must contain {COL_TIME} and {COL_SIG} columns")
+            if not (COL_VAL in data.columns or COL_VAL_TEXT in data.columns):
+                raise ValueError(f"DataFrame must contain at least one of {COL_VAL} or {COL_VAL_TEXT} columns")
+            value = (
+                pd.to_numeric(data[COL_VAL], errors="coerce") if COL_VAL in data.columns else pa.nulls(len(data))
+            )
+            value_text = data[COL_VAL_TEXT] if COL_VAL_TEXT in data.columns else pa.nulls(len(data))
+            table = pa.Table.from_arrays([data[COL_TIME], data[COL_SIG], value, value_text], schema=SCHEMA)
+
+        parquet_buffer = BytesIO()
+        pq.write_table(table, parquet_buffer)
+        parquet_buffer.seek(0)
+
+        files = {"file": ("data.parquet", parquet_buffer, "application/octet-stream")}
+        r = self._client.post(f"/stream/{self.datastream_id}/dataset/{self.id}/append", files=files)
+        validate_response(r, "Append data failed")
+
+    def cool(self) -> "Dataset":
+        """
+        Move all realtime data to cold storage and finalize this realtime dataset.
+
+        Cooling is started asynchronously on the server. Poll completion with
+        :meth:`wait_for_import`.
+
+        Only datasets in LIVE or COOLING_FAILED status can be cooled.
+        After cooling completes, the dataset no longer accepts appends and
+        ``import_status`` becomes FINISHED.
+
+        Returns:
+            The current dataset state (typically ``import_status == "COOLING"``).
+        """
+        if self.import_status not in ("LIVE", "COOLING_FAILED"):
+            raise ValueError(f"Dataset {self.id} cannot be cooled (status: {self.import_status})")
+
+        r = self._client.post(f"/stream/{self.datastream_id}/dataset/{self.id}/cool")
+        validate_response(r, "Cool dataset failed")
+        return self.fetch(self._client, self.id)
+
     def wait_for_import(self, timeout: float = 60, force_fetch: bool = False) -> "Dataset":
         """
-        Wait for the dataset import to complete.
+        Wait for the dataset import or cooling to complete.
 
-        If the dataset is still in a busy status (WAITING, IMPORTING, POST_PROCESSING, UPDATING_ICEBERG) after the timeout, a warning is issued and the current dataset information is returned.
-        If `force_fetch` is True, the import status is fetched at least once even if the dataset is not in a busy status, to ensure the latest status is returned.
+        If the dataset is still in a busy status (WAITING, IMPORTING, POST_PROCESSING,
+        UPDATING_ICEBERG, COOLING) after the timeout, a warning is issued and the current
+        dataset information is returned.
+        If `force_fetch` is True, the import status is fetched at least once even if the
+        dataset is not in a busy status, to ensure the latest status is returned.
         """
         if not (force_fetch or self.import_status in BUSY_STATUSES):
             return self
@@ -285,7 +368,9 @@ class DatasetList(UserList[Dataset]):
             try:
                 dataset = Dataset(client=client, **value)
             except ValidationError as e:
-                warnings.warn(f"Failed to create dataset with id {value.get('id')} and path {value.get('path')}: {e}")
+                warnings.warn(
+                    f"Failed to create dataset with id {value.get('id')} and path {value.get('path')}: {e}"
+                )
                 continue
             datasets.append(dataset)
         return cls(datasets)
@@ -296,7 +381,9 @@ class DatasetList(UserList[Dataset]):
         """
         return self.where(lambda d: d.import_status == "FINISHED")
 
-    def where_metadata(self, metadata: dict[str, int | str | Iterable[int | str]] | None = None) -> "DatasetList":
+    def where_metadata(
+        self, metadata: dict[str, int | str | Iterable[int | str]] | None = None
+    ) -> "DatasetList":
         """
         Filter datasets by their metadata fields.
 
@@ -510,3 +597,37 @@ class DatasetList(UserList[Dataset]):
             index=False,
         )
         return f"{df_str}\nDatasetList with {len(self.data)} datasets and {len(df.columns) - 5} unique metadata fields."
+
+
+def _detect_shape(shape: Optional[Literal["long", "wide"]], df: pd.DataFrame) -> Literal["long", "wide"]:
+    if shape is not None:
+        return shape
+
+    if "signal" in df.columns and ((COL_VAL in df.columns) or (COL_VAL_TEXT in df.columns)):
+        return "long"
+    return "wide"
+
+
+def _wide_to_long(df: pd.DataFrame) -> pa.Table:
+    signals = []
+    time = pa.array(df[COL_TIME], type=pa.int64())
+    for col in df.columns:
+        if col == COL_TIME:
+            continue
+        if (numeric_col := _to_numeric(df[col])) is not None:
+            value_arr = numeric_col.to_numpy().astype(np.float64)
+            value_text = pa.nulls(len(time))
+        else:
+            value_arr = pa.nulls(len(time))
+            value_text = df[col].fillna("").to_numpy().astype(str)
+        signals.append(pa.Table.from_arrays([time, [col] * len(time), value_arr, value_text], schema=SCHEMA))
+    return pa.concat_tables(signals)
+
+
+def _to_numeric(col: pd.Series) -> pd.Series | None:
+    if pd.api.types.is_numeric_dtype(col.dtype):
+        return col
+    null_count = col.isnull().sum()
+    numeric_col = pd.to_numeric(col, errors="coerce")
+    is_numeric = (numeric_col.isnull().sum() - null_count) / max(len(col), 1) < 0.2
+    return numeric_col if is_numeric else None

{marpledata-3.2.2 → marpledata-3.2.4}/src/marple/db/datastream.py

@@ -78,6 +78,22 @@ class DataStream(BaseModel):
         r = self._client.get(f"/stream/{self.id}/datasets")
         return DatasetList.from_dicts(self._client, validate_response(r, "Get datasets failed"))
 
+    def add_dataset(self, dataset_name: str, metadata: dict | None = None) -> Dataset:
+        """
+        Create a new empty dataset in this realtime stream.
+
+        Use :meth:`~marple.db.dataset.Dataset.append` to add data and
+        :meth:`~marple.db.dataset.Dataset.cool` to finalize the dataset.
+
+        To add datasets from a file to a file stream, use :meth:`push_file` instead.
+        """
+        r = self._client.post(
+            f"/stream/{self.id}/dataset/add",
+            json={"dataset_name": dataset_name, "metadata": metadata or {}},
+        )
+        r_json = validate_response(r, "Add dataset failed")
+        return self.get_dataset(r_json["dataset_id"])
+
     def push_file(
         self,
         file_path: str,

{marpledata-3.2.2 → marpledata-3.2.4}/src/marple/db/signal.py

@@ -1,12 +1,11 @@
+from pathlib import Path
 from typing import Literal
 
+import pandas as pd
 from pydantic import BaseModel, PrivateAttr
 
 from marple.utils import DBClient
 
-from pathlib import Path
-import pandas as pd
-
 
 class Signal(BaseModel):
     """
@@ -67,7 +66,9 @@ class Signal(BaseModel):
         """
         return self._client.list_parquet_files(self.dataset_id, self.id, refresh_cache)
 
-    def get_data(self, dtype: Literal["numeric", "text"] | None = None, refresh_cache: bool = False) -> pd.DataFrame:
+    def get_data(
+        self, dtype: Literal["numeric", "text"] | None = None, refresh_cache: bool = False
+    ) -> pd.DataFrame:
         """
         Get this signal's raw data as a pandas DataFrame.