marpledata 3.4.0.dev1__tar.gz → 3.5.0__tar.gz

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/.gitignore

@@ -169,4 +169,7 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 .idea/
-*.env
+.env*
+
+# Client handoff pack (may contain a real API token)
+/marple-matlab/

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/AGENTS.md

@@ -30,3 +30,4 @@ Run commands from `python/` unless a command explicitly says otherwise.
 - Integration tests run against live Marple services and may require
   `MDB_TOKEN`, `MDB_URL`, `INSIGHT_TOKEN`, and `INSIGHT_URL`. Tests should skip
   or fail clearly when required credentials are missing.
+- Release steps: `RELEASING.md`.

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/CHANGELOG.md

@@ -5,7 +5,18 @@ All notable changes to the Python SDK package `marpledata` will be documented in
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [3.4.0] - 2026-07-16
+## [3.5.0] - 2026-08-20
+
+### Added
+
+- `Dataset.add_signal` and `Dataset.add_signals` now accept a time-indexed pandas Series or DataFrame (DatetimeIndex / TimedeltaIndex when there is no `time` column).
+- `DataStream.push_file(..., overwrite=False)` to replace an existing dataset with the same name (also on the deprecated `DB.push_file`).
+
+### Fixed
+
+- An empty local parquet cache folder is treated as a cache miss and re-downloaded, instead of being assumed to mean the signal has no data.
+
+## [3.4.0] - 2026-07-28
 
 ### Added
 

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/CONTRIBUTING.md

@@ -55,34 +55,6 @@ Open `docs/_build/html/index.html` in your browser to view the site.
   - `marple.__version__`
   - `from marple import Insight, DB`
 
-### Publishing (Test Pypi)
+### Publishing
 
-Only once on your laptop
-
-- `uv build`
-- `uv publish --index testpypi`
-  - `username: __token__`
-  - `password: pypi-XXXXXXXXXXXXXXXXXXXXXXXXXXXX` (see 1Password)
-
-### Publishing Production
-
-- Make sure you don't have any local changes
-- Delete `./dist` folder
-- `uv version x.y.z(.devi)`
-- bump version in `__version__` variable
-- `uv build`
-- `uv publish` ⚠ **Impacts users, be careful**
-  - `username: __token__`
-  - `password: pypi-XXXXXXXXXXXXXXXXXXXXXXXXXXXX` (see 1Password)
-- Run the GitLab pipeline `pages` to build & release the docs
-
-**Versioning**
-
-To ensure pip correctly updates our package, correct versioning is important. Use `major.minor.patch`
-
-**Recommended test workflow**
-
-- Publish to test Pypi
-- Open a docker container with the desired python version: `docker run -it python:3.11-alpine sh`
-- Install the test pypi package `pip install --upgrade -i --pre https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple marpledata`
-- If you want to publish again, change the version to one that has not been used before. Even if you delete a build via [the UI](https://test.pypi.org/manage/project/marpledata/releases/) you cannot publish that version again. For testing, you could use something like `x.y.z.dev1`, `x.y.z.dev2`, `x.y.z.dev3`, ...
+See [`RELEASING.md`](RELEASING.md).

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: marpledata
-Version: 3.4.0.dev1
+Version: 3.5.0
 Summary: Marple SDK for Python
 Project-URL: Homepage, https://www.marpledata.com/
 Project-URL: Documentation, https://marpledata.gitlab.io/marple-sdk/
@@ -52,7 +52,7 @@ from marple import DB      # Marple DB
 from marple import Insight # Marple Insight
 ```
 
-For release notes, see [`CHANGELOG.md`](CHANGELOG.md).
+For release notes, see [`CHANGELOG.md`](CHANGELOG.md). To publish a release, see [`RELEASING.md`](RELEASING.md).
 
 ## Marple DB
 
@@ -90,34 +90,40 @@ dataset = dataset.wait_for_import(timeout=10)
 #### Add signals to an imported dataset
 
 Upload additional signals (for example derived channels) on an existing dataset.
-Each signal is a DataFrame, Arrow table, or on-disk parquet matching `LAKE_ARROW_SCHEMA`:
-columns `time` (int64 nanoseconds) plus `value` and/or `value_text`.
+DataFrames, Arrow tables, and on-disk parquet use `LAKE_ARROW_SCHEMA`: columns
+`time` (int64 nanoseconds) plus `value` and/or `value_text`.
 Times must overlap the dataset time range. Use `overwrite=True` to replace an existing signal name;
 a conflict without overwrite raises `SignalsAlreadyExistError`.
 
-```python
-import pandas as pd
-from marple.db import LAKE_ARROW_SCHEMA  # time + value and/or value_text
+A Series, or a DataFrame without a `time` column, takes its times from a `DatetimeIndex` or
+`TimedeltaIndex`. Indexed DataFrames must still have a `value` and/or `value_text` column.
 
-speed = dataset.get_signal("car.speed").get_data()
-derived = pd.DataFrame({
-    "time": speed.index.asi8,
-    "value": speed["value"] * 3.6,
-})
+```python
 # Single signal: wait until available before reading
+speed = dataset.get_signal("car.speed").get_data()
 signal = dataset.add_signal(
     "car.speed_kmh",
-    derived,
+    speed * 3.6,
     metadata={"unit": "km/h"},
 ).wait_until_available()
 
 # Batch: returns IDs as soon as upload completes (no wait)
 ids = dataset.add_signals([
-    {"name": "car.speed_kmh", "data": derived, "metadata": {"unit": "km/h"}},
+    {"name": "car.speed_kmh", "data": speed * 3.6, "metadata": {"unit": "km/h"}},
 ], overwrite=True, concurrency=4)
 signals = dataset.get_signals(signal_ids=ids)
 ```
 
+Data assembled from scratch uses the explicit schema columns instead:
+
+```python
+import pandas as pd
+from marple.db import LAKE_ARROW_SCHEMA  # time + value and/or value_text
+
+samples = pd.DataFrame({"time": [t0, t0 + 1_000_000_000], "value": [1.0, 2.0]})
+dataset.add_signal("car.custom", samples)
+```
+
 #### Upload large files
 
 `stream.push_file(...)` starts an ingestion and lets the Marple DB API choose the best upload mode. Depending on the deployment and file size, the SDK can upload through the API server, upload directly to Azure Blob Storage, use a single presigned URL, or split the file into multipart uploads.
@@ -203,7 +209,8 @@ if len(datasets) > 0:
 
 - **List streams**: `db.get_streams()`
 - **List datasets in a stream**: `stream.get_datasets()`
-- **Upload a file to a file-stream**: `stream.push_file(file_path, metadata={...}, concurrency=4)`
+- **Upload a file (file stream, default)**: `stream.push_file(file_path, metadata={...}, concurrency=4)`
+- **Custom lake ingest (file stream)**: `stream.add_dataset(...)` then `dataset.add_signal(...)` / `add_signals([...])`
 - **Add signals to a dataset**: `dataset.add_signal(name, data, ...)` / `dataset.add_signals([...])` — data matches `LAKE_ARROW_SCHEMA` (`time` + `value` and/or `value_text`)
 - **Wait until a signal is available**: `signal.wait_until_available(timeout=60)`
 - **Fetch signals by ID**: `dataset.get_signals(signal_ids=[...], refresh=False)`
@@ -214,12 +221,12 @@ if len(datasets) > 0:
 - **Delete a stream**: `stream.delete()` or `db.delete_stream(stream_key)`
 - **Delete a dataset**: `dataset.delete()` or `db.delete_dataset(dataset_id, dataset_path)`
 
-For live/realtime streams (creating and appending data):
+For live streams:
 
 - **Create an empty dataset**: `stream.add_dataset(dataset_name, metadata=None)`
 - **Upsert signal definitions**: `dataset.upsert_signals(signals=[...])`
 - **Append timeseries data**: `dataset.append(data=df, shape="long"|"wide"|None)`
-- **Cool a realtime dataset**: `dataset.cool()` then `dataset.wait_for_import()` to wait until `FINISHED`
+- **Cool to cold storage**: `dataset.cool()` then `dataset.wait_for_import()` until `FINISHED`
 
 ### Calling endpoints directly
 

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/README.md

@@ -17,7 +17,7 @@ from marple import DB      # Marple DB
 from marple import Insight # Marple Insight
 ```
 
-For release notes, see [`CHANGELOG.md`](CHANGELOG.md).
+For release notes, see [`CHANGELOG.md`](CHANGELOG.md). To publish a release, see [`RELEASING.md`](RELEASING.md).
 
 ## Marple DB
 
@@ -55,34 +55,40 @@ dataset = dataset.wait_for_import(timeout=10)
 #### Add signals to an imported dataset
 
 Upload additional signals (for example derived channels) on an existing dataset.
-Each signal is a DataFrame, Arrow table, or on-disk parquet matching `LAKE_ARROW_SCHEMA`:
-columns `time` (int64 nanoseconds) plus `value` and/or `value_text`.
+DataFrames, Arrow tables, and on-disk parquet use `LAKE_ARROW_SCHEMA`: columns
+`time` (int64 nanoseconds) plus `value` and/or `value_text`.
 Times must overlap the dataset time range. Use `overwrite=True` to replace an existing signal name;
 a conflict without overwrite raises `SignalsAlreadyExistError`.
 
-```python
-import pandas as pd
-from marple.db import LAKE_ARROW_SCHEMA  # time + value and/or value_text
+A Series, or a DataFrame without a `time` column, takes its times from a `DatetimeIndex` or
+`TimedeltaIndex`. Indexed DataFrames must still have a `value` and/or `value_text` column.
 
-speed = dataset.get_signal("car.speed").get_data()
-derived = pd.DataFrame({
-    "time": speed.index.asi8,
-    "value": speed["value"] * 3.6,
-})
+```python
 # Single signal: wait until available before reading
+speed = dataset.get_signal("car.speed").get_data()
 signal = dataset.add_signal(
     "car.speed_kmh",
-    derived,
+    speed * 3.6,
     metadata={"unit": "km/h"},
 ).wait_until_available()
 
 # Batch: returns IDs as soon as upload completes (no wait)
 ids = dataset.add_signals([
-    {"name": "car.speed_kmh", "data": derived, "metadata": {"unit": "km/h"}},
+    {"name": "car.speed_kmh", "data": speed * 3.6, "metadata": {"unit": "km/h"}},
 ], overwrite=True, concurrency=4)
 signals = dataset.get_signals(signal_ids=ids)
 ```
 
+Data assembled from scratch uses the explicit schema columns instead:
+
+```python
+import pandas as pd
+from marple.db import LAKE_ARROW_SCHEMA  # time + value and/or value_text
+
+samples = pd.DataFrame({"time": [t0, t0 + 1_000_000_000], "value": [1.0, 2.0]})
+dataset.add_signal("car.custom", samples)
+```
+
 #### Upload large files
 
 `stream.push_file(...)` starts an ingestion and lets the Marple DB API choose the best upload mode. Depending on the deployment and file size, the SDK can upload through the API server, upload directly to Azure Blob Storage, use a single presigned URL, or split the file into multipart uploads.
@@ -168,7 +174,8 @@ if len(datasets) > 0:
 
 - **List streams**: `db.get_streams()`
 - **List datasets in a stream**: `stream.get_datasets()`
-- **Upload a file to a file-stream**: `stream.push_file(file_path, metadata={...}, concurrency=4)`
+- **Upload a file (file stream, default)**: `stream.push_file(file_path, metadata={...}, concurrency=4)`
+- **Custom lake ingest (file stream)**: `stream.add_dataset(...)` then `dataset.add_signal(...)` / `add_signals([...])`
 - **Add signals to a dataset**: `dataset.add_signal(name, data, ...)` / `dataset.add_signals([...])` — data matches `LAKE_ARROW_SCHEMA` (`time` + `value` and/or `value_text`)
 - **Wait until a signal is available**: `signal.wait_until_available(timeout=60)`
 - **Fetch signals by ID**: `dataset.get_signals(signal_ids=[...], refresh=False)`
@@ -179,12 +186,12 @@ if len(datasets) > 0:
 - **Delete a stream**: `stream.delete()` or `db.delete_stream(stream_key)`
 - **Delete a dataset**: `dataset.delete()` or `db.delete_dataset(dataset_id, dataset_path)`
 
-For live/realtime streams (creating and appending data):
+For live streams:
 
 - **Create an empty dataset**: `stream.add_dataset(dataset_name, metadata=None)`
 - **Upsert signal definitions**: `dataset.upsert_signals(signals=[...])`
 - **Append timeseries data**: `dataset.append(data=df, shape="long"|"wide"|None)`
-- **Cool a realtime dataset**: `dataset.cool()` then `dataset.wait_for_import()` to wait until `FINISHED`
+- **Cool to cold storage**: `dataset.cool()` then `dataset.wait_for_import()` until `FINISHED`
 
 ### Calling endpoints directly
 

marpledata-3.5.0/RELEASING.md

@@ -0,0 +1,11 @@
+# Releasing `marpledata`
+
+Bump `pyproject.toml`, `__version__`, and `CHANGELOG.md` on `main` first.
+Tokens: 1Password (`username: __token__`). Run from `python/`.
+
+1. **TestPyPI** — `rm -rf dist && uv build && uv publish --index testpypi`  
+   Smoke: `pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple marpledata==<version>`
+2. **PyPI** — `rm -rf dist && uv build && uv publish`
+3. **Docs** — run the GitLab `pages` job (Sphinx → https://marpledata.gitlab.io/marple-sdk/)
+
+No GitHub tag/release; `CHANGELOG.md` is the release history.

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/docs/api/marple.db.Signal.rst

@@ -16,7 +16,7 @@
       ~Signal.cache_parquet
       ~Signal.get_data
       ~Signal.list_parquet_files
-      ~Signal.wait_until_cold
+      ~Signal.wait_until_available
    
    
 

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/docs/api/marple.db.SignalUpload.rst

@@ -26,7 +26,6 @@
       ~SignalUpload.name
       ~SignalUpload.data
       ~SignalUpload.metadata
-      ~SignalUpload.overwrite
       ~SignalUpload.priority
    
    

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/docs/getting-started.rst

@@ -37,8 +37,9 @@ Marple DB quickstart
    dataset = dataset.wait_for_import(timeout=10)
 
 After import, you can add derived signals with
-``dataset.add_signal(...)`` / ``dataset.add_signals([...])``. See
-:doc:`tutorials` for a full example.
+``dataset.add_signal(...)`` / ``dataset.add_signals([...])``. For custom
+ingest without file parsing, use ``stream.add_dataset(...)`` then
+``add_signal``. See :doc:`tutorials` for a full example.
 
 ``stream.push_file(...)`` starts an ingestion and lets the Marple DB API choose
 the best upload mode for the deployment and file size. For large files, use a

marpledata-3.5.0/docs/tutorials.rst

@@ -0,0 +1,125 @@
+Tutorials
+=========
+
+These examples use the high-level ``DataStream``, ``Dataset``, and ``Signal``
+APIs. Create a stream and API token in Marple DB first.
+
+Setup
+-----
+
+.. code-block:: python
+
+   import os
+   import re
+
+   import pandas as pd
+
+   from marple import DB
+
+   db = DB(os.environ["MDB_TOKEN"])
+   stream = db.get_stream("Car data")
+
+For VPC or self-hosted deployments, pass ``os.environ["MDB_URL"]`` as the
+second argument to ``DB``.
+
+Import a file and wait for import
+---------------------------------
+
+.. code-block:: python
+
+   dataset = stream.push_file(
+       "examples_race.csv",
+       metadata={"source": "testbench"},
+       concurrency=8,
+   ).wait_for_import(timeout=180)
+
+Add signals to a dataset
+------------------------
+
+Add signals after import, or start with an empty dataset using
+``stream.add_dataset``. Input follows :data:`marple.db.LAKE_ARROW_SCHEMA`:
+``time`` (int64 nanoseconds) plus ``value`` and/or ``value_text``.
+
+A Series, or a DataFrame without a ``time`` column, takes its times from a
+``DatetimeIndex`` or ``TimedeltaIndex``.
+
+.. code-block:: python
+
+   speed = dataset.get_signal("car.speed").get_data()
+   signal = dataset.add_signal(
+       "car.speed_kmh",
+       speed * 3.6,
+       metadata={"unit": "km/h"},
+   ).wait_until_available()
+
+Data assembled from scratch uses the explicit schema columns instead:
+
+.. code-block:: python
+
+   derived = pd.DataFrame({
+       "time": [t0, t0 + 1_000_000_000],
+       "value": [1.0, 2.0],
+   })
+   dataset.add_signal("car.custom", derived)
+
+For batches, ``add_signals`` returns signal IDs without waiting:
+
+.. code-block:: python
+
+   ids = dataset.add_signals(
+       [{"name": "car.speed_kmh", "data": speed * 3.6}],
+       overwrite=True,
+   )
+   signals = [
+       signal.wait_until_available()
+       for signal in dataset.get_signals(signal_ids=ids, refresh=True)
+   ]
+
+Filter datasets and get resampled data
+--------------------------------------
+
+.. code-block:: python
+
+   datasets = (
+       stream.get_datasets()
+       .where_metadata({"car_id": [1, 2], "track": "track_1"})
+       .wait_for_import()
+       .where_imported()
+       .where_signal("car.speed", "max", greater_than=75)
+   )
+   for dataset, data in datasets.get_data(
+       signals=["car.speed", re.compile(r"car\.wheel\..*\.speed")],
+       resample_rule="0.17s",
+   ):
+       print(dataset.path, data.shape)
+
+Ingest realtime data
+--------------------
+
+Use ``append`` with a realtime stream, then ``cool`` the dataset to cold
+storage when ingestion is complete.
+
+.. code-block:: python
+
+   realtime = db.get_stream("Live car data")
+   dataset = realtime.add_dataset("race-1", metadata={"driver": "Alice"})
+   dataset.upsert_signals([{"signal": "car.speed", "unit": "m/s"}])
+   dataset.append(pd.DataFrame({
+       "time": [1_700_000_000_000_000_000, 1_700_000_001_000_000_000],
+       "car.speed": [10.0, 12.0],
+   }))
+   dataset = dataset.cool().wait_for_import(timeout=180)
+
+Query with SQL
+--------------
+
+Trino querying is available on VPC and self-hosted deployments, not SaaS.
+
+.. code-block:: python
+
+   info = db.trino_info
+   table = f"{info['cold_catalog']}.{info['datapool']}.data"
+   data = db.query(
+       f"SELECT time, signal, value FROM {table} WHERE dataset = ? LIMIT 1000",
+       params=[dataset.id],
+   )

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/example.py

@@ -20,7 +20,9 @@ if stream_name not in [stream.name for stream in db.get_streams()]:
 
 stream = db.get_stream(stream_name)
 stream.push_file("examples_race.csv", metadata={"car_id": 1, "track": "track_1", "weather": "sunny"})
-stream.push_file("examples_race.csv", metadata={"car_id": 2, "track": "track_1", "weather": "cloudy"})
+stream.push_file(
+    "examples_race.csv", metadata={"car_id": 2, "track": "track_1", "weather": "cloudy"}, overwrite=True
+)
 stream.push_file("examples_race.csv", metadata={"car_id": 3, "track": "track_1", "weather": "rainy"})
 stream.push_file("examples_race.csv", metadata={"car_id": 1, "track": "track_2", "weather": "sunny"})
 stream.push_file("examples_race.csv", metadata={"car_id": 2, "track": "track_2", "weather": "sunny"})

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "marpledata"
-version = "3.4.0.dev1"
+version = "3.5.0"
 description = "Marple SDK for Python"
 authors = [
   { name = "Matthias Baert", email = "support@marpledata.com" },

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/src/marple/__init__.py

@@ -3,4 +3,4 @@ from .db import DB
 from .insight import Insight
 
 __all__ = ["DB", "Insight", "db"]
-__version__ = "3.4.0.dev1"
+__version__ = "3.5.0"

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/src/marple/db/__init__.py

@@ -353,19 +353,21 @@ class DB:
         file_path: str,
         metadata: dict | None = None,
         file_name: str | None = None,
+        overwrite: bool = False,
     ) -> int:
         """
         Push a file to a datastream.
         - `stream_key`: The name or ID of the stream to push the file to.
         - `file_path`: The path to the file to be pushed.
         - `metadata`: (optional) A dictionary of metadata to be associated with the file.
-        - `file_name`: (optional) The name of the file to be stored in the
+        - `file_name`: (optional) The name of the file to be stored in the stream.
+        - `overwrite`: (optional) If true, existing dataset with the same name will be overwritten.
 
         Note:
             This function is deprecated and it is encouraged to use the `push_file` method in the `DataStream` class directly.
         """
         stream = self.get_stream(stream_key)
-        return stream.push_file(file_path, metadata, file_name).id
+        return stream.push_file(file_path, metadata, file_name, overwrite=overwrite).id
 
     @deprecated
     def get_status(self, stream_key: str | int, dataset_id: int) -> dict:
@@ -451,17 +453,24 @@ class DB:
             metadata = {}
         self.get_dataset(dataset_id, dataset_path).update_metadata(metadata, overwrite)
 
-    # Realtime functions #
+    # Dataset creation and realtime ingest #
 
     def add_dataset(self, stream_key: str | int, dataset_name: str, metadata: dict | None = None) -> int:
         """
-        Create a new empty dataset in the specified live stream.
-        Returns the ID of the newly created dataset.
-
-        Use `dataset_append` to add data to the dataset and `upsert_signals` to define signals.
-        Use `cool` to finalize the dataset when done.
-
-        To add datasets from a file to a file stream, use `push_file` instead.
+        Create a new empty dataset in the specified stream.
+
+        **Live (realtime) stream**
+            - Define signals with :meth:`~marple.db.dataset.Dataset.upsert_signals`
+            - Push data with :meth:`~marple.db.dataset.Dataset.append`
+            - Call :meth:`~marple.db.dataset.Dataset.cool` when finished to move the
+              dataset to cold Parquet/Iceberg storage, then
+              :meth:`~marple.db.dataset.Dataset.wait_for_import` until it is `FINISHED`.
+        **File (non-live) stream**
+            - Default path: use :meth:`push_file` instead of `add_dataset` — upload a
+              file (e.g. CSV); Marple parses it and writes it to the data lake.
+            - Custom ingestion: create an empty dataset with `add_dataset`, then upload
+              signal data with :meth:`~marple.db.dataset.Dataset.add_signal` /
+              :meth:`~marple.db.dataset.Dataset.add_signals` straight into the lake.
         """
         return self.get_stream(stream_key).add_dataset(dataset_name, metadata).id
 

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/src/marple/db/dataset.py

@@ -333,24 +333,27 @@ class Dataset(BaseModel):
     def add_signal(
         self,
         name: str,
-        data: pa.Table | pd.DataFrame | Path | str,
+        data: pa.Table | pd.DataFrame | pd.Series | Path | str,
         metadata: dict[str, Any] | None = None,
         overwrite: bool = False,
         priority: Literal["default", "high"] = "default",
     ) -> Signal:
         """
-        Upload one signal with data into this dataset.
+        Upload one signal with data into this dataset (enrichment after import,
+        or custom ingest after :meth:`~marple.db.datastream.DataStream.add_dataset`).
 
-        ``data`` must be a DataFrame, Arrow table, or parquet path matching
+        ``data`` must be a DataFrame, Series, Arrow table, or parquet path matching
         :data:`~marple.db.LAKE_ARROW_SCHEMA` (``time`` plus ``value`` and/or
-        ``value_text``). Times must overlap the dataset time range.
+        ``value_text``). A Series, or a DataFrame without a ``time`` column, takes its
+        sample times from a ``DatetimeIndex`` or ``TimedeltaIndex``.
+        It must also contain ``value`` and/or ``value_text`` columns.
 
         Returns the new signal immediately after upload completes. Call
         :meth:`Signal.wait_until_available` to wait until the signal is available.
 
         Args:
             name: Signal name.
-            data: Signal samples (DataFrame, Arrow table, or parquet path).
+            data: Signal samples (DataFrame, Series, Arrow table, or parquet path).
             metadata: Optional signal metadata (for example ``unit``).
             overwrite: If True, replace an existing signal with the same name.
             priority: Import priority (``default`` or ``high``).

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/src/marple/db/datastream.py

@@ -79,12 +79,20 @@ class DataStream(BaseModel):
 
     def add_dataset(self, dataset_name: str, metadata: dict | None = None) -> Dataset:
         """
-        Create a new empty dataset in this realtime stream.
-
-        Use :meth:`~marple.db.dataset.Dataset.append` to add data and
-        :meth:`~marple.db.dataset.Dataset.cool` to finalize the dataset.
-
-        To add datasets from a file to a file stream, use :meth:`push_file` instead.
+        Create a new empty dataset in the specified stream.
+
+        **Live (realtime) stream**
+            - Define signals with :meth:`~marple.db.dataset.Dataset.upsert_signals`
+            - Push data with :meth:`~marple.db.dataset.Dataset.append`
+            - Call :meth:`~marple.db.dataset.Dataset.cool` when finished to move the
+              dataset to cold Parquet/Iceberg storage, then
+              :meth:`~marple.db.dataset.Dataset.wait_for_import` until it is `FINISHED`.
+        **File (non-live) stream**
+            - Default path: use :meth:`push_file` instead of `add_dataset` — upload a
+              file (e.g. CSV); Marple parses it and writes it to the data lake.
+            - Custom ingestion: create an empty dataset with `add_dataset`, then upload
+              signal data with :meth:`~marple.db.dataset.Dataset.add_signal` /
+              :meth:`~marple.db.dataset.Dataset.add_signals` straight into the lake.
         """
         r = self._client.post(
             f"/stream/{self.id}/dataset/add",
@@ -100,6 +108,7 @@ class DataStream(BaseModel):
         file_name: str | None = None,
         concurrency: int = 4,
         upload_mode: Literal["auto", "server"] = "auto",
+        overwrite: bool = False,
     ) -> Dataset:
         """
         Push a file to the datastream. The file will be ingested as a new dataset.
@@ -110,13 +119,14 @@ class DataStream(BaseModel):
             file_name: Optional name for the dataset. If not provided, the file name will be used.
             concurrency: Maximum number of concurrent part uploads for multipart uploads.
             upload_mode: Upload mode override. Use "server" to force upload through the Marple DB API server.
+            overwrite: If true, existing dataset with the same name will be overwritten.
         """
         if upload_mode not in ("auto", "server"):
             raise ValueError("upload_mode must be either 'auto' or 'server'")
 
         path = Path(file_path)
         file_size = path.stat().st_size
-        init = self._init_ingestion(file_name or path.name, file_size, metadata or {})
+        init = self._init_ingestion(file_name or path.name, file_size, metadata or {}, overwrite)
 
         try:
             if upload_mode == "server" or init.mode == "server":
@@ -137,7 +147,9 @@ class DataStream(BaseModel):
 
         return self.get_dataset(init.dataset_id)
 
-    def _init_ingestion(self, dataset_name: str, file_size: int, metadata: dict) -> IngestionInit:
+    def _init_ingestion(
+        self, dataset_name: str, file_size: int, metadata: dict, overwrite: bool = False
+    ) -> IngestionInit:
         r = self._client.post(
             "/ingestion",
             json={
@@ -145,6 +157,7 @@ class DataStream(BaseModel):
                 "dataset_name": dataset_name,
                 "file_size": file_size,
                 "metadata": metadata,
+                "overwrite": overwrite,
             },
         )
         return IngestionInit(**validate_response(r, "Initialize ingestion failed"))

{marpledata-3.4.0.dev1 → marpledata-3.5.0}/src/marple/db/signal.py

@@ -76,6 +76,7 @@ class Signal(BaseModel):
         Download the parquet files for this signal to a local cache folder and return the folder path.
         Args:
             refresh_cache: If True, re-download the parquet files even if they already exist in the cache.
+                An empty cache folder is always treated as a miss.
 
         Returns:
             The path to the local cache folder.
@@ -111,13 +112,11 @@ class Signal(BaseModel):
 
     def wait_until_available(self, timeout: float = 60) -> "Signal":
         """
-        Poll until this signal is available for querying (storage status is not
-        :attr:`StorageStatus.FROZEN_TO_COLD`). Updates the parent dataset's signal
-        cache when a parent dataset is known.
+        Poll until this signal is available for querying.
         Args:
-            timeout: Maximum time to wait for the signal to be available, in seconds.
+            timeout: Maximum time to wait for the signal to be available in seconds.
         Returns:
-            The signal object.
+            The signal object when it is available for querying, or a the last state before the timeout.
         """
         deadline_s = time.monotonic() + max(timeout, 0.1)
         while True: