marpledata 3.5.0__tar.gz → 3.6.0.dev1__tar.gz

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/AGENTS.md

@@ -15,6 +15,7 @@ This directory contains the Python SDK package published as `marpledata`.
 ## Commands
 
 - Run tests with output: `uv run pytest -vs`
+- Unit tests only: `uv run pytest -m "not integration"`
 - Build docs: `uv run --group docs sphinx-build -b html docs docs/_build/html`
 - Build package: `uv build`
 - Fix formatting: `uv run isort src tests && uv run black src tests`
@@ -29,5 +30,6 @@ Run commands from `python/` unless a command explicitly says otherwise.
 - Keep Python SDK tests in `python/tests/`.
 - Integration tests run against live Marple services and may require
   `MDB_TOKEN`, `MDB_URL`, `INSIGHT_TOKEN`, and `INSIGHT_URL`. Tests should skip
-  or fail clearly when required credentials are missing.
+  clearly when required credentials are missing. Mark live tests with
+  `pytest.mark.integration`.
 - Release steps: `RELEASING.md`.

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to the Python SDK package `marpledata` will be documented in
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Changed
+
+- Exact signal names in `get_data` / `get_signal` / `get_signals` resolve via the dataset cache or `GET /datapool/{pool}/signal/{name}/id` instead of downloading the full datapool `signal_map`. Regex patterns still use the map.
+
+### Added
+
+- `DB.delete_signals`, `Dataset.delete_signal` / `Dataset.delete_signals`, and `Signal.delete` to remove signals from a dataset.
+
+### Fixed
+
+- Allow `add_signal` in post-processing
+
 ## [3.5.0] - 2026-08-20
 
 ### Added

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/CONTRIBUTING.md

@@ -22,11 +22,10 @@ uv run mypy --install-types --non-interactive
 
 ### Testing
 
-The testing suite runs against a real linked DB & Insight deployment on our SaaS (e.g. Castro Comrades).
-Among other things it will create a stream, ingest a dataset, and export that dataset from Insight
-Upload tests also exercise server and multipart upload flows, and create/delete temporary streams with a test prefix.
+The testing suite includes local unit tests and integration tests against a live Marple DB / Insight deployment. Integration tests share one imported example dataset per session, use a small CSV for extra ingestions, and mark themselves with `pytest.mark.integration`.
 
 ```bash
+uv run pytest -m "not integration"   # local unit tests only
 export MDB_TOKEN=...
 export INSIGHT_TOKEN=...
 # Optional, defaults to SaaS URLs:

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: marpledata
-Version: 3.5.0
+Version: 3.6.0.dev1
 Summary: Marple SDK for Python
 Project-URL: Homepage, https://www.marpledata.com/
 Project-URL: Documentation, https://marpledata.gitlab.io/marple-sdk/
@@ -220,6 +220,7 @@ if len(datasets) > 0:
 - **Get a resampled df of multiple signals**: `dataset.get_data(signals=[...], resample_rule="1s")`
 - **Delete a stream**: `stream.delete()` or `db.delete_stream(stream_key)`
 - **Delete a dataset**: `dataset.delete()` or `db.delete_dataset(dataset_id, dataset_path)`
+- **Delete signals**: `signal.delete()`, `dataset.delete_signal(signal_id)` / `dataset.delete_signals(signal_ids)`, or `db.delete_signals(dataset_id, dataset_path, signal_ids)`
 
 For live streams:
 

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/README.md

@@ -185,6 +185,7 @@ if len(datasets) > 0:
 - **Get a resampled df of multiple signals**: `dataset.get_data(signals=[...], resample_rule="1s")`
 - **Delete a stream**: `stream.delete()` or `db.delete_stream(stream_key)`
 - **Delete a dataset**: `dataset.delete()` or `db.delete_dataset(dataset_id, dataset_path)`
+- **Delete signals**: `signal.delete()`, `dataset.delete_signal(signal_id)` / `dataset.delete_signals(signal_ids)`, or `db.delete_signals(dataset_id, dataset_path, signal_ids)`
 
 For live streams:
 

marpledata-3.6.0.dev1/RELEASING.md

@@ -0,0 +1,13 @@
+# Releasing `marpledata`
+
+Tokens: 1Password (`username: __token__`). Run from `python/`.
+
+- **Bump version**
+  - `uv version --bump minor` / `uv version x.y.z`
+  - Manual update `__init__.py`, `CHANGELOG.md`
+- **TestPyPI** — `rm -rf dist && uv build && uv publish --index testpypi`
+  Smoke: `pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple marpledata==<version>`
+- **PyPI** — `rm -rf dist && uv build && uv publish`
+- **Docs** — run the GitLab `pages` job (Sphinx → https://marpledata.gitlab.io/marple-sdk/)
+
+No GitHub tag/release; `CHANGELOG.md` is the release history.

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/docs/getting-started.rst

@@ -50,6 +50,13 @@ longer ``wait_for_import`` timeout and optionally increase upload concurrency:
    dataset = stream.push_file("large_export.csv", concurrency=8)
    dataset = dataset.wait_for_import(timeout=180)
 
+To replace an existing dataset with the same name, pass ``overwrite=True``:
+
+.. code-block:: python
+
+   dataset = stream.push_file("examples_race.csv", overwrite=True)
+   dataset = dataset.wait_for_import(timeout=10)
+
 If direct storage uploads are blocked by your network or proxy, force upload
 through the Marple DB API server:
 

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/docs/tutorials.rst

@@ -31,8 +31,11 @@ Import a file and wait for import
        "examples_race.csv",
        metadata={"source": "testbench"},
        concurrency=8,
+       overwrite=False,
    ).wait_for_import(timeout=180)
 
+Pass ``overwrite=True`` to replace an existing dataset with the same name.
+
 Add signals to a dataset
 ------------------------
 

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "marpledata"
-version = "3.5.0"
+version = "3.6.0.dev1"
 description = "Marple SDK for Python"
 authors = [
   { name = "Matthias Baert", email = "support@marpledata.com" },
@@ -72,6 +72,11 @@ docs = [
   "sphinx>=7.2.6",
 ]
 
+[tool.pytest.ini_options]
+markers = [
+  "integration: tests that talk to a live Marple DB / Insight instance",
+]
+
 [tool.black]
 line-length = 113
 target-version = ["py311"]

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/src/marple/__init__.py

@@ -3,4 +3,4 @@ from .db import DB
 from .insight import Insight
 
 __all__ = ["DB", "Insight", "db"]
-__version__ = "3.5.0"
+__version__ = "3.6.0.dev1"

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/src/marple/db/__init__.py

@@ -432,6 +432,20 @@ class DB:
         r = self.post(f"/stream/{dataset.datastream_id}/dataset/{dataset.id}/delete")
         validate_response(r, "Delete dataset failed")
 
+    def delete_signals(
+        self,
+        dataset_id: int | None,
+        dataset_path: str | None,
+        signal_ids: Sequence[int],
+    ) -> None:
+        """
+        Delete signals from a dataset by ID or path.
+
+        Warning:
+            This is a destructive operation that cannot be undone.
+        """
+        self.get_dataset(dataset_id, dataset_path).delete_signals(signal_ids)
+
     @deprecated
     def update_metadata(
         self,

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/src/marple/db/dataset.py

@@ -131,7 +131,7 @@ class Dataset(BaseModel):
         """Get a specific signal in this dataset by its name or ID.
 
         Args:
-            name: Signal name (resolved via the datapool signal map).
+            name: Signal name.
             id: Signal ID.
             refresh: If True, refetch from the API even when cached.
         """
@@ -141,7 +141,7 @@ class Dataset(BaseModel):
             raise ValueError("Only one of name or id can be provided.")
 
         if name is not None:
-            id = self._client.get_signal_map().get(name)
+            id = self._client.get_signal_id(name)
 
         if id is None:
             raise ValueError(f"Signal with name {name} not found in dataset with id {self.id}.")
@@ -192,19 +192,39 @@ class Dataset(BaseModel):
         """
         if signal_names is not None and signal_ids is not None:
             raise ValueError("Provide only one of signal_names or signal_ids")
-
-        if signal_ids is not None:
-            return self._get_signals_by_ids(list(signal_ids), refresh=refresh)
-
-        if signal_names is None:
+        if signal_ids is None and signal_names is None:
             return self._get_all_signals()
-
-        matched = self._client.find_matching_signals(signal_names)
-        if not matched:
-            return []
-        return self._get_signals_by_ids(list(matched.values()), refresh=refresh)
-
-    def _get_signals_by_ids(self, signal_ids: Sequence[int], *, refresh: bool = False) -> list["Signal"]:
+        if signal_ids is None:
+            assert signal_names is not None
+            signal_ids = self._find_signal_ids(signal_names).values()
+        return self._get_signals_by_ids(list(signal_ids), refresh=refresh)
+
+    def _find_signal_ids(self, signals: Iterable[str | re.Pattern]) -> dict[str, int]:
+        found: dict[str, int] = {}
+        exact: list[str] = []
+        patterns: list[re.Pattern] = []
+        for signal in signals:
+            (exact if isinstance(signal, str) else patterns).append(signal)  # type: ignore [arg-type]
+
+        if patterns:
+            # Only look up the signal map in the regex case.
+            for name, sig_id in self._client.get_signal_map().items():
+                if any(pattern.search(name) for pattern in patterns):
+                    found[name] = sig_id
+
+        if self.n_signals is not None and self.n_signals < 100:
+            self._get_all_signals()
+
+        cached = {s.name: s.id for s in self._signals.values()}
+        for name in exact:
+            if (signal_id := cached.get(name)) is not None:
+                found[name] = signal_id
+            elif (signal_id := self._client.get_signal_id(name)) is not None:
+                found[name] = signal_id
+
+        return found
+
+    def _get_signals_by_ids(self, signal_ids: list[int], *, refresh: bool = False) -> list["Signal"]:
         if not signal_ids:
             return []
 
@@ -254,7 +274,7 @@ class Dataset(BaseModel):
             A pandas DataFrame containing one column per signal, aligned on time.
         """
         return self._get_signals_dataframe(
-            self._client.find_matching_signals(signals).items(),
+            self._find_signal_ids(signals).items(),
             resample_rule,
             resample_aggregate,
             dtype,
@@ -390,7 +410,7 @@ class Dataset(BaseModel):
             return []
         if len(signals) > MAX_SIGNALS_PER_ADD:
             raise ValueError(f"Provide at most {MAX_SIGNALS_PER_ADD} signals per call")
-        if self.import_status != ImportStatus.FINISHED:
+        if self.import_status not in (ImportStatus.FINISHED, ImportStatus.POSTPROCESSING):
             raise ValueError(f"Dataset {self.id} is not in a writable state (status: {self.import_status})")
 
         signal_ids = run_signal_uploads(self, signals, overwrite=overwrite, concurrency=concurrency)
@@ -488,6 +508,33 @@ class Dataset(BaseModel):
         warnings.warn(f"Import did not finish after {timeout} seconds")
         return self.fetch(self._client, self.id)
 
+    def delete_signal(self, signal_id: int) -> None:
+        """
+        Delete a single signal from this dataset by ID.
+
+        Warning:
+            This is a destructive action that cannot be undone.
+        """
+        self.delete_signals([signal_id])
+
+    def delete_signals(self, signal_ids: Sequence[int]) -> None:
+        """
+        Delete one or more signals from this dataset by ID.
+
+        Warning:
+            This is a destructive action that cannot be undone.
+        """
+        to_delete = list(set(signal_ids))
+        if not to_delete:
+            return
+        r = self._client.post(
+            f"/stream/{self.datastream_id}/dataset/{self.id}/signals/delete",
+            json={"signal_ids": to_delete},
+        )
+        validate_response(r, "Delete signals failed")
+        for signal_id in to_delete:
+            self._signals.pop(signal_id, None)
+
     def delete(self) -> None:
         """
         Delete the dataset.
@@ -691,7 +738,7 @@ class DatasetList(UserList[Dataset]):
         if len(self.data) == 0:
             return
         # Avoid having to search signals for every individual dataset
-        signal_pairs = list(self.data[0]._client.find_matching_signals(signals).items())
+        signal_pairs = list(self.data[0]._find_signal_ids(signals).items())
         for dataset in self.data:
             yield dataset, dataset._get_signals_dataframe(
                 signals=signal_pairs,

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/src/marple/db/signal.py

@@ -137,3 +137,16 @@ class Signal(BaseModel):
                 warnings.warn(f"Signal {self.name} did not reach a queryable state after {timeout} seconds")
                 return fresh
             time.sleep(0.5)
+
+    def delete(self) -> None:
+        """
+        Delete this signal from its dataset.
+
+        Warning:
+            This is a destructive action that cannot be undone.
+        """
+        r = self._client.post(
+            f"/stream/{self.datastream_id}/dataset/{self.dataset_id}/signals/delete",
+            json={"signal_ids": [self.id]},
+        )
+        validate_response(r, "Delete signal failed")

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/src/marple/db/signal_upload.py

@@ -286,7 +286,7 @@ def _presign_signals(
             body = {}
         raise SignalsAlreadyExistError(
             body.get("signals") or [],  # type: ignore[arg-type]
-            message=f"Signal upload failed: {body.get('error', 'signals_already_exist')}",
+            message=f"Signal upload failed (use overwrite=True to replace existing signals): {body.get('error', 'signals_already_exist')}",
         )
     signals = {
         item["name"]: PresignedSignal.model_validate(item)

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/src/marple/utils.py

@@ -1,6 +1,5 @@
-import re
 from pathlib import Path
-from typing import Any, Iterable, Literal
+from typing import Any, Literal
 from urllib import parse, request
 
 import pandas as pd
@@ -130,17 +129,13 @@ class DBClient:
             self._signal_map = validate_response(r, "Get signals failed")
         return self._signal_map
 
-    def find_matching_signals(self, signals: Iterable[str | re.Pattern]) -> dict[str, int]:
-        all_signals = self.get_signal_map()
-        matching = dict()
-        for pattern in signals:
-            if isinstance(pattern, str) and pattern in all_signals:
-                matching[pattern] = all_signals[pattern]
-            elif isinstance(pattern, re.Pattern):
-                for name, id in all_signals.items():
-                    if pattern.search(name):
-                        matching[name] = id
-        return matching
+    def get_signal_id(self, name: str) -> int | None:
+        if self._signal_map is not None:
+            return self._signal_map.get(name)
+        r = self.get(f"/datapool/{self.datapool}/signal/{parse.quote(name, safe='')}/id")
+        if r.status_code == 404:
+            return None
+        return validate_response(r, "Get signal id failed")["id"]
 
     def cache_parquet(self, dataset_id: int, signal_id: int, refresh_cache: bool = False) -> Path:
         """

{marpledata-3.5.0 → marpledata-3.6.0.dev1}/tests/conftest.py

@@ -16,11 +16,11 @@ dotenv.load_dotenv()
 def _required_env(name: str) -> str:
     value = os.getenv(name)
     if value is None:
-        pytest.fail(f"Missing env var {name}; skipping integration test.")
+        pytest.skip(f"Missing env var {name}; skipping integration test.")
     return value
 
 
-@pytest.fixture()
+@pytest.fixture(scope="session")
 def db() -> DB:
     url = os.getenv("MDB_URL", marple.db.SAAS_URL)
     assert url is not None
@@ -35,17 +35,13 @@ def insight() -> Insight:
 
 
 @pytest.fixture(scope="session")
-def example_stream() -> Generator[DataStream, None, None]:
-    url = os.getenv("MDB_URL", marple.db.SAAS_URL)
-    assert url is not None
-    session_db = DB(_required_env("MDB_TOKEN"), url)
-
+def example_stream(db: DB) -> Generator[DataStream, None, None]:
     name = "Salty Compulsory Pytest " + datetime.now().isoformat()
-    yield session_db.create_stream(name)
+    yield db.create_stream(name)
     print("Cleaning up stream...")
-    session_db.delete_stream(name)
+    db.delete_stream(name)
 
 
-@pytest.fixture()
+@pytest.fixture(scope="session")
 def example_dataset(example_stream: DataStream) -> Dataset:
     return ingest_dataset(example_stream, metadata={"A": 1, "B": 1})

marpledata-3.6.0.dev1/tests/support.py

@@ -0,0 +1,60 @@
+from collections.abc import Iterator
+from contextlib import contextmanager
+from datetime import datetime
+from pathlib import Path
+from uuid import uuid4
+
+import marple
+from marple import DB
+from marple.db import Dataset, DataStream
+
+EXAMPLE_CSV = Path(__file__).parents[2] / "test_data" / "examples_race.csv"
+
+_swept_prefixes: set[str] = set()
+
+
+def unique_name(prefix: str, suffix: str = "") -> str:
+    """Datapool-unique name so parallel Python/Rust CI jobs cannot collide."""
+    return f"{prefix}-{uuid4().hex}{suffix}"
+
+
+def sweep_streams_by_prefix(db: DB, prefix: str) -> None:
+    """Delete leftover streams with this prefix once per process."""
+    if prefix in _swept_prefixes:
+        return
+    _swept_prefixes.add(prefix)
+    for stream in db.get_streams():
+        if stream.name.startswith(prefix):
+            try:
+                db.delete_stream(stream.id)
+            except Exception:
+                pass
+
+
+@contextmanager
+def isolated_stream(db: DB, prefix: str, suffix: str, **create_kwargs) -> Iterator[DataStream]:
+    sweep_streams_by_prefix(db, prefix)
+    stream = db.create_stream(f"{prefix} {suffix} {datetime.now().isoformat()}", **create_kwargs)
+    try:
+        yield stream
+    finally:
+        try:
+            db.delete_stream(stream.id)
+        except Exception:
+            pass
+
+
+def ingest_dataset(stream: DataStream, metadata: dict | None = None) -> Dataset:
+    dataset = stream.push_file(
+        str(EXAMPLE_CSV),
+        metadata={
+            "source": "pytest:test_db.py",
+            "sdk_version": marple.__version__,
+        }
+        | (metadata or {}),
+        file_name=unique_name("py-sdk", ".csv"),
+    ).wait_for_import(timeout=180)
+    assert (
+        dataset.import_status == "FINISHED"
+    ), f"Dataset {dataset.id} did not finish importing (status: {dataset.import_status})"
+    return dataset