marpledata 3.6.0.dev1__tar.gz → 3.7.0__tar.gz

{marpledata-3.6.0.dev1 → marpledata-3.7.0}/CHANGELOG.md

@@ -5,15 +5,31 @@ All notable changes to the Python SDK package `marpledata` will be documented in
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [3.7.0] - 2026-09-28
+
+### Changed
+
+- Clarify docs to use add_signals() for bulk adding signals
+- `Dataset.add_signal` and `Dataset.add_signals` can upload onto a dataset in `POSTPROCESSING_FAILED`
+
+## [3.6.0] - 2026-09-04
 
 ### Changed
 
 - Exact signal names in `get_data` / `get_signal` / `get_signals` resolve via the dataset cache or `GET /datapool/{pool}/signal/{name}/id` instead of downloading the full datapool `signal_map`. Regex patterns still use the map.
+- Optional `plugin_args` on `DataStream.push_file` (and the deprecated `DB.push_file` wrapper).
 
 ### Added
 
-- `DB.delete_signals`, `Dataset.delete_signal` / `Dataset.delete_signals`, and `Signal.delete` to remove signals from a dataset.
+- Processing Pipeline
+  - `DB.create_script` / `get_scripts` / `get_script` / `delete_script`
+  - `Script.update` / `duplicate` / `delete`, to manage stored `process(dataset)` scripts.
+  - `Dataset.run` / `DB.run_script` to execute a stored processing script against a dataset via a server-side sandbox job.
+  - `DataStream.scripts`, `DataStream.update`, and `DataStream.rerun_processing` (`DB.rerun_processing`) to edit a stream and set the script pipeline
+  - `Dataset.rerun_processing` and `Dataset.get_debug_messages`.
+  - `DB.delete_signals`, `Dataset.delete_signal` / `Dataset.delete_signals`, and `Signal.delete` to remove signals from a dataset.
+  - `Dataset.reingest` to reingest a dataset from its original uploaded file, with optional `plugin_args`.
+  - `logger.debug` lines on SDK mutations (`add_signal`, `update_metadata`, …), off by default. Call `db.verbose()` to print them.
 
 ### Fixed
 

{marpledata-3.6.0.dev1 → marpledata-3.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: marpledata
-Version: 3.6.0.dev1
+Version: 3.7.0
 Summary: Marple SDK for Python
 Project-URL: Homepage, https://www.marpledata.com/
 Project-URL: Documentation, https://marpledata.gitlab.io/marple-sdk/
@@ -78,6 +78,7 @@ API_TOKEN = "<your api token>"
 API_URL = "https://db.marpledata.com/api/v1"  # optional if using the default SaaS
 
 db = DB(API_TOKEN, API_URL)
+db.verbose()  # print SDK mutation activity (push_file, add_signal, …) to stdout
 
 db.check_connection()
 
@@ -98,6 +99,8 @@ a conflict without overwrite raises `SignalsAlreadyExistError`.
 A Series, or a DataFrame without a `time` column, takes its times from a `DatetimeIndex` or
 `TimedeltaIndex`. Indexed DataFrames must still have a `value` and/or `value_text` column.
 
+Warning: for performance reasons, prefer adding signals in bulk using `add_signals` over multiple usages of `add_signal`.
+
 ```python
 # Single signal: wait until available before reading
 speed = dataset.get_signal("car.speed").get_data()
@@ -124,6 +127,42 @@ samples = pd.DataFrame({"time": [t0, t0 + 1_000_000_000], "value": [1.0, 2.0]})
 dataset.add_signal("car.custom", samples)
 ```
 
+#### Processing scripts
+
+Write a `process(dataset)` function, store it, and try it on any imported dataset. This runs on the server and writes to that dataset.
+Warning: for performance reasons, prefer adding signals in bulk using `add_signals` over multiple usages of `add_signal`.
+
+```python
+source = """
+from marple.db import Dataset
+
+def process(dataset: Dataset) -> None:
+    speed = dataset.get_signal("car.speed").get_data()
+    dataset.add_signal("car.speed_kmh", speed * 3.6, metadata={"unit": "km/h"})
+"""
+
+script = db.create_script("speed_kmh", source)
+dataset = stream.get_dataset(path="lap.csv")
+# or: dataset = stream.push_file("lap.csv").wait_for_import()
+dataset.run(script)
+```
+
+Pass a `.py` path or `pathlib.Path` to `create_script` / `script.update` instead of source text. Iterate with `script.update(script=...)` then `dataset.run(script)` again (or `dataset.run(script, source=...)` to save and run in one step). New signals written by the script may need `signal.wait_until_available(...)` before you read them.
+
+When the script looks right, attach it to the stream with `stream.update(scripts=...)`. That **replaces** the pipeline (pass `[]` to detach all). New uploads then run those scripts after ingest.
+
+```python
+stream = stream.update(scripts=[script.id])
+```
+
+For files already imported, rerun aliasing and the stream's script pipeline:
+
+```python
+dataset = dataset.rerun_processing().wait_for_import()
+```
+
+Use `script.update(...)` to change source or metadata, and `dataset.get_debug_messages()` for the latest ingestion's debug log (not the sandbox job log from `dataset.run`).
+
 #### Upload large files
 
 `stream.push_file(...)` starts an ingestion and lets the Marple DB API choose the best upload mode. Depending on the deployment and file size, the SDK can upload through the API server, upload directly to Azure Blob Storage, use a single presigned URL, or split the file into multipart uploads.
@@ -221,6 +260,11 @@ if len(datasets) > 0:
 - **Delete a stream**: `stream.delete()` or `db.delete_stream(stream_key)`
 - **Delete a dataset**: `dataset.delete()` or `db.delete_dataset(dataset_id, dataset_path)`
 - **Delete signals**: `signal.delete()`, `dataset.delete_signal(signal_id)` / `dataset.delete_signals(signal_ids)`, or `db.delete_signals(dataset_id, dataset_path, signal_ids)`
+- **Run a processing script**: `dataset.run(script)` or `db.run_script(dataset_id, script)`
+- **Create a processing script**: `db.create_script(name, script)`
+- **Set script pipeline** (replaces the full list): `stream.update(scripts=[script.id])`
+- **Rerun aliasing + scripts**: `dataset.rerun_processing().wait_for_import()` or `stream.rerun_processing([dataset.id])`
+- **Read ingest debug logs**: `dataset.get_debug_messages()`
 
 For live streams:
 

{marpledata-3.6.0.dev1 → marpledata-3.7.0}/README.md

@@ -43,6 +43,7 @@ API_TOKEN = "<your api token>"
 API_URL = "https://db.marpledata.com/api/v1"  # optional if using the default SaaS
 
 db = DB(API_TOKEN, API_URL)
+db.verbose()  # print SDK mutation activity (push_file, add_signal, …) to stdout
 
 db.check_connection()
 
@@ -63,6 +64,8 @@ a conflict without overwrite raises `SignalsAlreadyExistError`.
 A Series, or a DataFrame without a `time` column, takes its times from a `DatetimeIndex` or
 `TimedeltaIndex`. Indexed DataFrames must still have a `value` and/or `value_text` column.
 
+Warning: for performance reasons, prefer adding signals in bulk using `add_signals` over multiple usages of `add_signal`.
+
 ```python
 # Single signal: wait until available before reading
 speed = dataset.get_signal("car.speed").get_data()
@@ -89,6 +92,42 @@ samples = pd.DataFrame({"time": [t0, t0 + 1_000_000_000], "value": [1.0, 2.0]})
 dataset.add_signal("car.custom", samples)
 ```
 
+#### Processing scripts
+
+Write a `process(dataset)` function, store it, and try it on any imported dataset. This runs on the server and writes to that dataset.
+Warning: for performance reasons, prefer adding signals in bulk using `add_signals` over multiple usages of `add_signal`.
+
+```python
+source = """
+from marple.db import Dataset
+
+def process(dataset: Dataset) -> None:
+    speed = dataset.get_signal("car.speed").get_data()
+    dataset.add_signal("car.speed_kmh", speed * 3.6, metadata={"unit": "km/h"})
+"""
+
+script = db.create_script("speed_kmh", source)
+dataset = stream.get_dataset(path="lap.csv")
+# or: dataset = stream.push_file("lap.csv").wait_for_import()
+dataset.run(script)
+```
+
+Pass a `.py` path or `pathlib.Path` to `create_script` / `script.update` instead of source text. Iterate with `script.update(script=...)` then `dataset.run(script)` again (or `dataset.run(script, source=...)` to save and run in one step). New signals written by the script may need `signal.wait_until_available(...)` before you read them.
+
+When the script looks right, attach it to the stream with `stream.update(scripts=...)`. That **replaces** the pipeline (pass `[]` to detach all). New uploads then run those scripts after ingest.
+
+```python
+stream = stream.update(scripts=[script.id])
+```
+
+For files already imported, rerun aliasing and the stream's script pipeline:
+
+```python
+dataset = dataset.rerun_processing().wait_for_import()
+```
+
+Use `script.update(...)` to change source or metadata, and `dataset.get_debug_messages()` for the latest ingestion's debug log (not the sandbox job log from `dataset.run`).
+
 #### Upload large files
 
 `stream.push_file(...)` starts an ingestion and lets the Marple DB API choose the best upload mode. Depending on the deployment and file size, the SDK can upload through the API server, upload directly to Azure Blob Storage, use a single presigned URL, or split the file into multipart uploads.
@@ -186,6 +225,11 @@ if len(datasets) > 0:
 - **Delete a stream**: `stream.delete()` or `db.delete_stream(stream_key)`
 - **Delete a dataset**: `dataset.delete()` or `db.delete_dataset(dataset_id, dataset_path)`
 - **Delete signals**: `signal.delete()`, `dataset.delete_signal(signal_id)` / `dataset.delete_signals(signal_ids)`, or `db.delete_signals(dataset_id, dataset_path, signal_ids)`
+- **Run a processing script**: `dataset.run(script)` or `db.run_script(dataset_id, script)`
+- **Create a processing script**: `db.create_script(name, script)`
+- **Set script pipeline** (replaces the full list): `stream.update(scripts=[script.id])`
+- **Rerun aliasing + scripts**: `dataset.rerun_processing().wait_for_import()` or `stream.rerun_processing([dataset.id])`
+- **Read ingest debug logs**: `dataset.get_debug_messages()`
 
 For live streams:
 

{marpledata-3.6.0.dev1 → marpledata-3.7.0}/docs/api/db.rst

@@ -14,6 +14,10 @@ See :doc:`../tutorials` for upload examples.
    marple.db.DataStream
    marple.db.Dataset
    marple.db.DatasetList
+   marple.db.SandboxJob
+   marple.db.SandboxJobStatus
+   marple.db.Script
+   marple.db.ScriptVersion
    marple.db.Signal
    marple.db.SignalUpload
    marple.db.SignalsAlreadyExistError

{marpledata-3.6.0.dev1 → marpledata-3.7.0}/docs/getting-started.rst

@@ -40,6 +40,7 @@ After import, you can add derived signals with
 ``dataset.add_signal(...)`` / ``dataset.add_signals([...])``. For custom
 ingest without file parsing, use ``stream.add_dataset(...)`` then
 ``add_signal``. See :doc:`tutorials` for a full example.
+Warning: for performance reasons, prefer adding signals in bulk using ``add_signals`` over multiple usages of ``add_signal``.
 
 ``stream.push_file(...)`` starts an ingestion and lets the Marple DB API choose
 the best upload mode for the deployment and file size. For large files, use a

{marpledata-3.6.0.dev1 → marpledata-3.7.0}/docs/tutorials.rst

@@ -46,6 +46,8 @@ Add signals after import, or start with an empty dataset using
 A Series, or a DataFrame without a ``time`` column, takes its times from a
 ``DatetimeIndex`` or ``TimedeltaIndex``.
 
+Warning: for performance reasons, prefer adding signals in bulk using ``add_signals`` over multiple usages of ``add_signal``.
+
 .. code-block:: python
 
    speed = dataset.get_signal("car.speed").get_data()
@@ -78,6 +80,45 @@ For batches, ``add_signals`` returns signal IDs without waiting:
        for signal in dataset.get_signals(signal_ids=ids, refresh=True)
    ]
 
+Processing scripts
+------------------
+
+Write a ``process(dataset)`` function, store it, and try it on any imported dataset.
+This runs on the server and writes to that dataset.
+Warning: for performance reasons, prefer adding signals in bulk using ``add_signals`` over multiple usages of ``add_signal``.
+
+.. code-block:: python
+
+   source = """
+   from marple.db import Dataset
+
+   def process(dataset: Dataset) -> None:
+       speed = dataset.get_signal("car.speed").get_data()
+       dataset.add_signal("car.speed_kmh", speed * 3.6, metadata={"unit": "km/h"})
+   """
+
+   script = db.create_script("speed_kmh", source)
+   dataset = stream.get_dataset(path="lap.csv")
+   dataset.run(script)
+
+Pass a ``.py`` path instead of source text. Iterate with
+``script.update(script=...)`` then ``dataset.run(script)`` (or
+``dataset.run(script, source=...)`` to save and run in one step).
+
+When the script looks right, attach it to the stream with
+``stream.update(scripts=...)``. That **replaces** the pipeline
+(pass ``[]`` to detach all). New uploads then run those scripts after ingest.
+
+.. code-block:: python
+
+   stream = stream.update(scripts=[script.id])
+
+For files already imported, rerun aliasing and the stream's script pipeline:
+
+.. code-block:: python
+
+   dataset = dataset.rerun_processing().wait_for_import()
+
 Filter datasets and get resampled data
 --------------------------------------
 

{marpledata-3.6.0.dev1 → marpledata-3.7.0}/example.py

@@ -2,6 +2,7 @@ import os
 import re
 
 import dotenv
+
 from src.marple.db import DB, Dataset
 
 dotenv.load_dotenv()
@@ -11,6 +12,7 @@ if api_token is None:
 
 url = os.getenv("MDB_URL")
 db = DB(api_token, url)
+db.verbose()  # print SDK mutation activity (push_file, add_signal, …) to stdout
 db.check_connection()
 
 stream_name = "CSV Stream"

{marpledata-3.6.0.dev1 → marpledata-3.7.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "marpledata"
-version = "3.6.0.dev1"
+version = "3.7.0"
 description = "Marple SDK for Python"
 authors = [
   { name = "Matthias Baert", email = "support@marpledata.com" },

{marpledata-3.6.0.dev1 → marpledata-3.7.0}/src/marple/__init__.py

@@ -3,4 +3,4 @@ from .db import DB
 from .insight import Insight
 
 __all__ = ["DB", "Insight", "db"]
-__version__ = "3.6.0.dev1"
+__version__ = "3.7.0"

{marpledata-3.6.0.dev1 → marpledata-3.7.0}/src/marple/db/__init__.py

@@ -1,19 +1,21 @@
 import warnings
 from functools import wraps
 from pathlib import Path
-from typing import TYPE_CHECKING, Any, Literal, Optional, Sequence
+from typing import TYPE_CHECKING, Any, Literal, Optional, Sequence, TextIO
 
 import pandas as pd
 from pydantic import ValidationError
 from requests import Response
 from requests.exceptions import ConnectionError
 
-from marple.db import sql
+from marple.db import activity, sql
+from marple.db.activity import logger
 from marple.db.constants import LAKE_ARROW_SCHEMA as _LAKE_ARROW_SCHEMA
 from marple.db.constants import SAAS_URL
 from marple.db.constants import SCHEMA as _SCHEMA
 from marple.db.dataset import Dataset, DatasetList
 from marple.db.datastream import DataStream
+from marple.db.script import SandboxJob, SandboxJobStatus, Script, ScriptVersion
 from marple.db.signal import Signal
 from marple.db.signal_upload import SignalsAlreadyExistError, SignalUpload
 from marple.utils import DBClient, validate_response
@@ -37,6 +39,10 @@ __all__ = [
     "DataStream",
     "Dataset",
     "DatasetList",
+    "SandboxJob",
+    "SandboxJobStatus",
+    "Script",
+    "ScriptVersion",
     "Signal",
     "SignalUpload",
     "SignalsAlreadyExistError",
@@ -151,6 +157,10 @@ class DB:
         self._refresh_stream_cache(r)
         return True
 
+    def verbose(self, enabled: bool = True, file: TextIO | None = None) -> None:
+        """Turn SDK mutation activity logging on or off (stdout by default)."""
+        activity.verbose(enabled, file)
+
     # Stream functions #
 
     def create_stream(
@@ -237,6 +247,92 @@ class DB:
             f"Stream with name or id {stream_key} not found, available streams: {', '.join([s.name for s in self._streams.values()])}"
         )
 
+    def rerun_processing(self, stream_key: str | int, dataset_ids: Sequence[int] | None = None) -> None:
+        """
+        Rerun aliasing and processing scripts for a stream or selected datasets.
+
+        See :meth:`~marple.db.datastream.DataStream.rerun_processing`.
+        """
+        self.get_stream(stream_key).rerun_processing(dataset_ids)
+
+    def get_scripts(self) -> list[Script]:
+        """List processing scripts in this workspace (without source / version history)."""
+        r = self.get("/scripts")
+        return [
+            Script(client=self.client, **script)
+            for script in validate_response(r, "Get scripts failed")["scripts"]
+        ]
+
+    def get_script(self, script_id: int) -> Script:
+        """Get a processing script by ID, including recent versions and source."""
+        return Script.fetch(self.client, script_id)
+
+    def run_script(
+        self,
+        dataset_id: int,
+        script: Script | int,
+        *,
+        source: str | Path | None = None,
+        version: int | None = None,
+        timeout: float = 180,
+    ) -> Dataset:
+        """Run a stored processing script on a dataset.
+
+        See :meth:`~marple.db.dataset.Dataset.run`.
+
+        Args:
+            dataset_id: The ID of the dataset to run the script on.
+            script: A stored script or its ID.
+            source: Optional source text or ``.py`` path to save before running.
+            version: Script version ID to run. Defaults to the latest version.
+            timeout: Seconds to wait for the sandbox job to finish.
+
+        Warning:
+            This is not a dry-run. The script will be executed and the dataset will be modified.
+        """
+        return self.get_dataset(dataset_id).run(script, source=source, version=version, timeout=timeout)
+
+    def create_script(
+        self,
+        name: str,
+        script: str | Path,
+        *,
+        description: str | None = None,
+    ) -> Script:
+        """
+        Create a processing script.
+
+        Attach it to a stream with :meth:`~marple.db.datastream.DataStream.update`
+        (``scripts=`` replaces the full pipeline).
+
+        Args:
+            name: The name of the script.
+            script: Source text or a path to a file that must define ``process(dataset)``.
+            description: The description of the script.
+        """
+        r = self.post(
+            "/script",
+            json={
+                "name": name,
+                "description": description,
+                "script": Script.resolve_source(script),
+            },
+        )
+        created = Script(client=self.client, **validate_response(r, "Create script failed"))
+        logger.debug(f"Created script {name}")
+        return created
+
+    def delete_script(self, script_id: int) -> None:
+        """
+        Delete a processing script.
+
+        Warning:
+            This cannot be undone. The script is removed from every stream pipeline.
+        """
+        r = self.delete(f"/script/{script_id}")
+        validate_response(r, "Delete script failed")
+        logger.debug(f"Deleted script {script_id}")
+
     def _refresh_stream_cache(self, r: Response | None = None) -> None:
         if r is None:
             r = self.get("/streams")
@@ -354,6 +450,7 @@ class DB:
         metadata: dict | None = None,
         file_name: str | None = None,
         overwrite: bool = False,
+        plugin_args: str | None = None,
     ) -> int:
         """
         Push a file to a datastream.
@@ -362,12 +459,13 @@ class DB:
         - `metadata`: (optional) A dictionary of metadata to be associated with the file.
         - `file_name`: (optional) The name of the file to be stored in the stream.
         - `overwrite`: (optional) If true, existing dataset with the same name will be overwritten.
+        - `plugin_args`: (optional) Plugin arguments for this ingest.
 
         Note:
             This function is deprecated and it is encouraged to use the `push_file` method in the `DataStream` class directly.
         """
         stream = self.get_stream(stream_key)
-        return stream.push_file(file_path, metadata, file_name, overwrite=overwrite).id
+        return stream.push_file(file_path, metadata, file_name, overwrite=overwrite, plugin_args=plugin_args).id
 
     @deprecated
     def get_status(self, stream_key: str | int, dataset_id: int) -> dict:

marpledata-3.7.0/src/marple/db/activity.py

@@ -0,0 +1,28 @@
+import logging
+import sys
+from typing import TextIO
+
+logger = logging.getLogger("marple.sdk")
+logger.addHandler(logging.NullHandler())
+logger.propagate = False
+
+_handler: logging.Handler | None = None
+
+
+def verbose(enabled: bool = True, file: TextIO | None = None) -> None:
+    """Turn SDK mutation activity logging on or off (stdout by default)."""
+    global _handler
+    if enabled:
+        logger.setLevel(logging.DEBUG)
+        if _handler is not None:
+            return
+        _handler = logging.StreamHandler(file or sys.stdout)
+        _handler.setFormatter(logging.Formatter("%(message)s"))
+        logger.addHandler(_handler)
+        return
+
+    if _handler is not None:
+        logger.removeHandler(_handler)
+        _handler.close()
+        _handler = None
+    logger.setLevel(logging.NOTSET)