mainsequence 4.3.18__tar.gz → 4.3.20__tar.gz

{mainsequence-4.3.18/mainsequence.egg-info → mainsequence-4.3.20}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.3.18
+Version: 4.3.20
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement

{mainsequence-4.3.18 → mainsequence-4.3.20}/agent_scaffold/skills/data_publishing/data_nodes/SKILL.md

@@ -335,7 +335,80 @@ Use `UpdateStatistics`.
 Do not fetch or return full history every run unless there is a documented
 reason.
 
-### 6. `time_index` Must Be Nanosecond UTC
+### 6. DataNode Tail Deletes Must Use `delete_after_date(...)`
+
+When a DataNode workflow needs to remove persisted rows from its
+`PlatformTimeIndexMetaTable` storage table, use
+`TimeIndexMetaTable.delete_after_date(...)`. This is the only normal DataNode
+storage deletion path.
+
+Do not use raw SQL, compiled SQL operations, direct database clients,
+`run_query(...)`, backend-private endpoints, table truncation, or ad hoc
+delete helpers to clean DataNode storage rows.
+
+The SDK call targets:
+
+```text
+POST /orm/api/ts_manager/dynamic_table/<dynamic_table_uid>/delete_after_date/
+```
+
+Use a global tail delete only when all streams in the table should be rolled
+back from the same inclusive cutoff:
+
+```python
+storage = MyStorageTable.get_time_index_meta_table()
+storage.delete_after_date("2026-04-01T00:00:00Z")
+```
+
+For multidimensional storage, prefer an explicit scope. Use
+`dimension_filters` when deleting all rows for one or more dimension values:
+
+```python
+storage.delete_after_date(
+    "2026-04-01T00:00:00Z",
+    dimension_filters={
+        "asset_identifier": ["example-asset-btc", "example-asset-eth"],
+    },
+)
+```
+
+Use `index_coordinates` when deleting exact coordinate streams:
+
+```python
+storage.delete_after_date(
+    "2026-04-01T00:00:00Z",
+    index_coordinates=[
+        {
+            "account_uid": "account-a",
+            "asset_identifier": "example-asset-btc",
+        }
+    ],
+)
+```
+
+Deleting all rows for scoped streams is allowed only with an explicit scope:
+
+```python
+storage.delete_after_date(
+    None,
+    dimension_filters={"asset_identifier": ["example-asset-btc"]},
+)
+
+storage.delete_after_date(
+    None,
+    index_coordinates=[
+        {
+            "account_uid": "account-a",
+            "asset_identifier": "example-asset-btc",
+        }
+    ],
+)
+```
+
+Never send `after_date=None` without `dimension_filters` or
+`index_coordinates`. That is an unbounded table delete and must be rejected.
+
+### 7. `time_index` Must Be Nanosecond UTC
 
 Every non-empty DataFrame returned by `update()` must have its first index
 level named `time_index` with dtype exactly `datetime64[ns, UTC]`.
@@ -386,7 +459,7 @@ physical schema evolution. Keep the `PlatformTimeIndexMetaTable` catalog model a
 the SDK storage contract, apply Alembic-rendered SQL through the migration
 workflow, then register or refresh the MetaTable catalog binding separately.
 
-### 7. Dependencies Must Be Deterministic
+### 8. Dependencies Must Be Deterministic
 
 Dependencies belong in constructor setup and `dependencies()`.
 
@@ -395,7 +468,7 @@ changing it changes the dependency graph and update identity.
 
 Do not construct dependency graphs dynamically inside `update()`.
 
-### 8. Foreign Keys Belong To SQLAlchemy And Alembic
+### 9. Foreign Keys Belong To SQLAlchemy And Alembic
 
 For new code, model foreign keys on the `PlatformTimeIndexMetaTable` storage
 class, or route the storage work to the MetaTable skill. When a DataNode storage
@@ -416,7 +489,7 @@ binding after upgrade.
 
 Do not add DataNode configuration fields just to mutate storage metadata.
 
-### 9. Metadata Belongs To Storage
+### 10. Metadata Belongs To Storage
 
 Production-quality table identifiers, descriptions, labels, column docs, and
 foreign-key metadata belong to the storage class/MetaTable registration path.
@@ -444,6 +517,8 @@ When reviewing an existing DataNode, look for:
 - misuse of `hash_namespace`
 - non-incremental `update()` behavior
 - hidden dependency creation inside `update()`
+- DataNode storage cleanup that bypasses `TimeIndexMetaTable.delete_after_date(...)`
+- `delete_after_date(None)` without explicit `dimension_filters` or `index_coordinates`
 - invalid identity-indexed output shape
 - `time_index` dtype that is not exactly `datetime64[ns, UTC]`
 - DataFrame columns that do not match the `PlatformTimeIndexMetaTable` class
@@ -464,6 +539,9 @@ Do not claim success until you have checked:
 - no `test_node` usage remains
 - `dependencies()` is deterministic
 - `update()` is incremental
+- DataNode storage deletion, rollback, and repair paths use
+  `TimeIndexMetaTable.delete_after_date(...)`
+- unbounded deletes with `after_date=None` and no dimension/coordinate scope are absent
 - the DataFrame shape matches the storage class
 - non-empty outputs have first index level `time_index` with dtype `datetime64[ns, UTC]`
 - the first validation run uses explicit `hash_namespace(...)` when it touches a shared backend

{mainsequence-4.3.18 → mainsequence-4.3.20}/agent_scaffold/skills/data_publishing/meta_tables/SKILL.md

@@ -387,6 +387,24 @@ Only use physical table names returned by registered `MetaTable` objects when co
 
 Do not hardcode platform-managed physical names manually.
 
+### 7. DataNode storage deletes use the DataNode tail-delete API
+
+For `PlatformTimeIndexMetaTable` storage owned by DataNodes, do not design raw
+SQL delete operations or compiled SQL delete operations for rollback, repair, or
+stream cleanup. Route that work to the DataNode skill and use
+`TimeIndexMetaTable.delete_after_date(...)`.
+
+The DataNode delete path is:
+
+```text
+POST /orm/api/ts_manager/dynamic_table/<dynamic_table_uid>/delete_after_date/
+```
+
+Use `after_date` for global tail rollback. Use `dimension_filters` or
+`index_coordinates` for scoped deletes, including scoped full-stream deletes
+where `after_date=None`. Never allow `after_date=None` without an explicit
+dimension or coordinate scope.
+
 ## Review Rules
 
 When reviewing an existing MetaTable workflow, look for:
@@ -406,6 +424,8 @@ When reviewing an existing MetaTable workflow, look for:
 - migration work that asks users to define backend payloads, artifact rows, or SDK request objects
 - compiled SQL operations without complete table scope
 - raw SQL that hardcodes stale physical names
+- raw SQL or compiled SQL deletes against DataNode-owned
+  `PlatformTimeIndexMetaTable` storage instead of `delete_after_date(...)`
 - a table that should really be modeled as a DataNode instead
 
 ## Validation Checklist

{mainsequence-4.3.18 → mainsequence-4.3.20}/mainsequence/cli/cli.py

@@ -11450,6 +11450,66 @@ def project_sync(
     success(f"Synced: {repo_name}")
 
 
+@project.command("sync-after-commit")
+def project_sync_after_commit(
+    project_uid: str | None = typer.Argument(
+        None,
+        help="Project UID. If omitted, read MAIN_SEQUENCE_PROJECT_UID from local .env.",
+    ),
+    path: str | None = typer.Option(
+        None,
+        "--path",
+        help="Project directory used to resolve MAIN_SEQUENCE_PROJECT_UID when PROJECT_UID is omitted.",
+    ),
+    timeout: int | None = typer.Option(None, "--timeout", help="Request timeout in seconds."),
+):
+    """
+    Trigger backend post-commit project sync.
+
+    This directly calls:
+
+    POST /orm/api/pods/projects/<project_uid>/sync_project_after_commit/
+
+    Examples
+    --------
+    ```bash
+    mainsequence project sync-after-commit project-uid-123
+    mainsequence project sync-after-commit --path .
+    ```
+    """
+    _require_login()
+
+    resolved_project_uid = project_uid
+    if resolved_project_uid is None:
+        project_dir = normalize_path(path) if path else pathlib.Path.cwd()
+        if path and not project_dir.exists():
+            error(f"Folder does not exist: {project_dir}")
+            raise typer.Exit(1)
+        resolved_project_uid = _read_project_ref_from_env_file(project_dir)
+
+    if resolved_project_uid is None:
+        error(
+            "Could not determine project uid. Pass PROJECT_UID or ensure "
+            "MAIN_SEQUENCE_PROJECT_UID is present in local .env."
+        )
+        raise typer.Exit(1)
+
+    try:
+        payload = sync_project_after_commit(resolved_project_uid, timeout=timeout)
+    except ApiError as e:
+        error(f"Backend post-commit sync failed: {e}")
+        raise typer.Exit(1) from e
+
+    result = payload or {
+        "project_uid": resolved_project_uid,
+        "detail": "sync_project_after_commit triggered",
+    }
+    if _emit_json(result):
+        return
+
+    success(f"Triggered backend sync for project {resolved_project_uid}.")
+
+
 @project.command("sync_project", hidden=True)
 def project_sync_project(
     message: str = typer.Argument(..., help="Git commit message"),

{mainsequence-4.3.18 → mainsequence-4.3.20}/mainsequence/client/metatables/core.py

@@ -3027,6 +3027,56 @@ class TimeIndexMetaTable(MetaTable):
                     descriptor[key] = cls._date_for_payload(descriptor[key])
         return normalized
 
+    def _identity_dimensions_from_time_indexed_profile(self) -> list[str]:
+        profile = self.time_indexed_profile
+        if profile is None:
+            raise ValueError(
+                "Cannot validate dimension_range_map because TimeIndexMetaTable "
+                "is missing time_indexed_profile."
+            )
+
+        time_index_name = str(profile.time_index_name)
+        index_names = [str(name) for name in profile.index_names]
+        if not time_index_name or not index_names:
+            raise ValueError(
+                "Cannot validate dimension_range_map because TimeIndexMetaTable "
+                "time_indexed_profile is missing time_index_name or index_names."
+            )
+        return [name for name in index_names if name != time_index_name]
+
+    def _validate_dimension_range_map_coordinates(
+        self,
+        dimension_range_map: list[dict[str, Any]] | None,
+    ) -> None:
+        if not dimension_range_map:
+            return
+
+        expected_dimensions = self._identity_dimensions_from_time_indexed_profile()
+        if not expected_dimensions:
+            return
+
+        for position, descriptor in enumerate(dimension_range_map):
+            coordinate = descriptor.get("coordinate") if isinstance(descriptor, Mapping) else None
+            provided_dimensions = (
+                [str(name) for name in coordinate.keys()]
+                if isinstance(coordinate, Mapping)
+                else []
+            )
+            provided_set = set(provided_dimensions)
+            missing_dimensions = [
+                dimension
+                for dimension in expected_dimensions
+                if dimension not in provided_set
+            ]
+            if missing_dimensions:
+                raise ValueError(
+                    "dimension_range_map coordinate is incomplete for "
+                    "TimeIndexMetaTable read. "
+                    f"Entry {position} expected identity dimensions "
+                    f"{expected_dimensions!r}; provided {provided_dimensions!r}; "
+                    f"missing {missing_dimensions!r}."
+                )
+
     def _build_dimension_payload(
         self,
         *,
@@ -3040,6 +3090,7 @@ class TimeIndexMetaTable(MetaTable):
         if index_coordinates is not None:
             payload["index_coordinates"] = index_coordinates
         if dimension_range_map is not None:
+            self._validate_dimension_range_map_coordinates(dimension_range_map)
             payload["dimension_range_map"] = self._normalize_dimension_range_map(
                 dimension_range_map
             )
@@ -3326,7 +3377,7 @@ class TimeIndexMetaTable(MetaTable):
                 )
                 if r.status_code != 200:
                     logger.warning(f"Error in request: {r.text}")
-                    return [], None
+                    raise_for_response(r, payload=payload)
 
                 response_data = r.json()
                 # Accumulate results

{mainsequence-4.3.18 → mainsequence-4.3.20}/mainsequence/client/models_foundry.py

@@ -183,6 +183,16 @@ class Project(LabelableObjectMixin, ShareableObjectMixin, BasePydanticModel, Bas
         examples=["git@github.com:mainsequence/data-pipeline.git"],
         json_schema_extra={"label": "Git SSH URL"},
     )
+    latest_git_version: str = Field(
+        "",
+        title="Latest Git Version",
+        description=(
+            "Normalized highest valid version extracted from repository tags for the "
+            "project's configured branch. Empty when the backend has not found a valid version."
+        ),
+        examples=["1.2.3"],
+        json_schema_extra={"label": "Latest Git Version"},
+    )
     created_by: str | int | dict[str, Any] | None = Field(
         None,
         title="Created By",

{mainsequence-4.3.18 → mainsequence-4.3.20/mainsequence.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.3.18
+Version: 4.3.20
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement

{mainsequence-4.3.18 → mainsequence-4.3.20}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mainsequence"
-version = "4.3.18"
+version = "4.3.20"
 description = "Main Sequence SDK "
 readme = "README.md"
 requires-python = ">=3.11"

{mainsequence-4.3.18 → mainsequence-4.3.20}/tests/test_cli.py

@@ -10294,6 +10294,59 @@ def test_project_sync_triggers_backend_sync_after_push(cli_mod, runner, monkeypa
     assert "Triggered backend sync for project project-uid-123." in result.output
 
 
+def test_project_sync_after_commit_uses_project_uid(cli_mod, runner, monkeypatch):
+    captured = {}
+
+    monkeypatch.setattr(cli_mod, "_require_login", lambda: {"username": "u"})
+    monkeypatch.setattr(
+        cli_mod,
+        "sync_project_after_commit",
+        lambda project_uid, timeout=None: captured.update(
+            project_uid=project_uid,
+            timeout=timeout,
+        )
+        or {"uid": project_uid},
+    )
+
+    result = runner.invoke(
+        cli_mod.app,
+        ["project", "sync-after-commit", "project-uid-123", "--timeout", "30"],
+    )
+
+    assert result.exit_code == 0
+    assert captured == {"project_uid": "project-uid-123", "timeout": 30}
+    assert "Triggered backend sync for project project-uid-123." in result.output
+
+
+def test_project_sync_after_commit_reads_project_uid_from_env_file(
+    cli_mod, runner, monkeypatch, tmp_path
+):
+    target = tmp_path / "project"
+    target.mkdir()
+    (target / ".env").write_text("MAIN_SEQUENCE_PROJECT_UID=project-uid-123\n", encoding="utf-8")
+
+    monkeypatch.setattr(cli_mod, "_require_login", lambda: {"username": "u"})
+    monkeypatch.setattr(
+        cli_mod,
+        "sync_project_after_commit",
+        lambda project_uid, timeout=None: {
+            "project_uid": project_uid,
+            "status": "queued",
+        },
+    )
+
+    result = runner.invoke(
+        cli_mod.app,
+        ["project", "sync-after-commit", "--path", str(target), "--json"],
+    )
+
+    assert result.exit_code == 0
+    assert json.loads(result.output) == {
+        "project_uid": "project-uid-123",
+        "status": "queued",
+    }
+
+
 def test_project_sync_project(cli_mod, runner, monkeypatch, tmp_path):
     target = tmp_path / "project"
     target.mkdir(parents=True, exist_ok=True)

{mainsequence-4.3.18 → mainsequence-4.3.20}/tests/test_data_node_storage_dimension_queries.py

@@ -1,8 +1,10 @@
 import datetime
 
 import pandas as pd
+import pytest
 
 from mainsequence.client import metatables as models_metatables
+from mainsequence.client.exceptions import BadRequestError
 
 
 def _source_config(index_names: list[str]) -> models_metatables.TimeIndexedProfile:
@@ -173,6 +175,79 @@ def test_get_data_between_dates_from_api_sends_dimension_range_map(monkeypatch):
     ]
 
 
+def test_get_data_between_dates_rejects_incomplete_dimension_range_map(monkeypatch):
+    def _fake_make_request(**_kwargs):
+        raise AssertionError("request should not be sent with an incomplete coordinate")
+
+    monkeypatch.setattr(models_metatables, "make_request", _fake_make_request)
+
+    start = datetime.datetime(2026, 5, 1, 0, tzinfo=datetime.UTC)
+
+    with pytest.raises(ValueError) as exc:
+        _storage(
+            ["time_index", "account_uid", "asset_identifier"]
+        ).get_data_between_dates_from_api(
+            start_date=start,
+            dimension_range_map=[
+                {
+                    "coordinate": {"account_uid": "account-a"},
+                    "start_date": start,
+                }
+            ],
+        )
+
+    message = str(exc.value)
+    assert "dimension_range_map coordinate is incomplete" in message
+    assert "asset_identifier" in message
+    assert "account_uid" in message
+
+
+def test_get_data_between_dates_raises_backend_error(monkeypatch):
+    class FakeResponse:
+        status_code = 400
+        text = '{"error": {"dimension_range_map": ["Index coordinate missing dimension(s): asset_identifier"]}}'
+
+        @staticmethod
+        def json():
+            return {
+                "error": {
+                    "dimension_range_map": [
+                        "Index coordinate missing dimension(s): asset_identifier"
+                    ]
+                }
+            }
+
+    def _fake_make_request(*, s, loaders, payload, r_type, url):
+        return FakeResponse()
+
+    monkeypatch.setattr(models_metatables, "make_request", _fake_make_request)
+    monkeypatch.setattr(
+        models_metatables.TimeIndexMetaTable,
+        "build_session",
+        classmethod(lambda cls: object()),
+    )
+
+    start = datetime.datetime(2026, 5, 1, 0, tzinfo=datetime.UTC)
+
+    with pytest.raises(BadRequestError) as exc:
+        _storage(
+            ["time_index", "account_uid", "asset_identifier"]
+        ).get_data_between_dates_from_api(
+            start_date=start,
+            dimension_range_map=[
+                {
+                    "coordinate": {
+                        "account_uid": "account-a",
+                        "asset_identifier": "BTC",
+                    },
+                    "start_date": start,
+                }
+            ],
+        )
+
+    assert "asset_identifier" in str(exc.value)
+
+
 def test_get_data_between_dates_from_node_identifier_sends_canonical_dimensions(monkeypatch):
     captured = {}