mainsequence 4.3.19__tar.gz → 4.3.22__tar.gz

{mainsequence-4.3.19/mainsequence.egg-info → mainsequence-4.3.22}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.3.19
+Version: 4.3.22
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement

{mainsequence-4.3.19 → mainsequence-4.3.22}/agent_scaffold/skills/data_publishing/data_nodes/SKILL.md

@@ -335,7 +335,80 @@ Use `UpdateStatistics`.
 Do not fetch or return full history every run unless there is a documented
 reason.
 
-### 6. `time_index` Must Be Nanosecond UTC
+### 6. DataNode Tail Deletes Must Use `delete_after_date(...)`
+
+When a DataNode workflow needs to remove persisted rows from its
+`PlatformTimeIndexMetaTable` storage table, use
+`TimeIndexMetaTable.delete_after_date(...)`. This is the only normal DataNode
+storage deletion path.
+
+Do not use raw SQL, compiled SQL operations, direct database clients,
+`run_query(...)`, backend-private endpoints, table truncation, or ad hoc
+delete helpers to clean DataNode storage rows.
+
+The SDK call targets:
+
+```text
+POST /orm/api/ts_manager/dynamic_table/<dynamic_table_uid>/delete_after_date/
+```
+
+Use a global tail delete only when all streams in the table should be rolled
+back from the same inclusive cutoff:
+
+```python
+storage = MyStorageTable.get_time_index_meta_table()
+storage.delete_after_date("2026-04-01T00:00:00Z")
+```
+
+For multidimensional storage, prefer an explicit scope. Use
+`dimension_filters` when deleting all rows for one or more dimension values:
+
+```python
+storage.delete_after_date(
+    "2026-04-01T00:00:00Z",
+    dimension_filters={
+        "asset_identifier": ["example-asset-btc", "example-asset-eth"],
+    },
+)
+```
+
+Use `index_coordinates` when deleting exact coordinate streams:
+
+```python
+storage.delete_after_date(
+    "2026-04-01T00:00:00Z",
+    index_coordinates=[
+        {
+            "account_uid": "account-a",
+            "asset_identifier": "example-asset-btc",
+        }
+    ],
+)
+```
+
+Deleting all rows for scoped streams is allowed only with an explicit scope:
+
+```python
+storage.delete_after_date(
+    None,
+    dimension_filters={"asset_identifier": ["example-asset-btc"]},
+)
+
+storage.delete_after_date(
+    None,
+    index_coordinates=[
+        {
+            "account_uid": "account-a",
+            "asset_identifier": "example-asset-btc",
+        }
+    ],
+)
+```
+
+Never send `after_date=None` without `dimension_filters` or
+`index_coordinates`. That is an unbounded table delete and must be rejected.
+
+### 7. `time_index` Must Be Nanosecond UTC
 
 Every non-empty DataFrame returned by `update()` must have its first index
 level named `time_index` with dtype exactly `datetime64[ns, UTC]`.
@@ -386,7 +459,7 @@ physical schema evolution. Keep the `PlatformTimeIndexMetaTable` catalog model a
 the SDK storage contract, apply Alembic-rendered SQL through the migration
 workflow, then register or refresh the MetaTable catalog binding separately.
 
-### 7. Dependencies Must Be Deterministic
+### 8. Dependencies Must Be Deterministic
 
 Dependencies belong in constructor setup and `dependencies()`.
 
@@ -395,7 +468,7 @@ changing it changes the dependency graph and update identity.
 
 Do not construct dependency graphs dynamically inside `update()`.
 
-### 8. Foreign Keys Belong To SQLAlchemy And Alembic
+### 9. Foreign Keys Belong To SQLAlchemy And Alembic
 
 For new code, model foreign keys on the `PlatformTimeIndexMetaTable` storage
 class, or route the storage work to the MetaTable skill. When a DataNode storage
@@ -416,7 +489,7 @@ binding after upgrade.
 
 Do not add DataNode configuration fields just to mutate storage metadata.
 
-### 9. Metadata Belongs To Storage
+### 10. Metadata Belongs To Storage
 
 Production-quality table identifiers, descriptions, labels, column docs, and
 foreign-key metadata belong to the storage class/MetaTable registration path.
@@ -444,6 +517,8 @@ When reviewing an existing DataNode, look for:
 - misuse of `hash_namespace`
 - non-incremental `update()` behavior
 - hidden dependency creation inside `update()`
+- DataNode storage cleanup that bypasses `TimeIndexMetaTable.delete_after_date(...)`
+- `delete_after_date(None)` without explicit `dimension_filters` or `index_coordinates`
 - invalid identity-indexed output shape
 - `time_index` dtype that is not exactly `datetime64[ns, UTC]`
 - DataFrame columns that do not match the `PlatformTimeIndexMetaTable` class
@@ -464,6 +539,9 @@ Do not claim success until you have checked:
 - no `test_node` usage remains
 - `dependencies()` is deterministic
 - `update()` is incremental
+- DataNode storage deletion, rollback, and repair paths use
+  `TimeIndexMetaTable.delete_after_date(...)`
+- unbounded deletes with `after_date=None` and no dimension/coordinate scope are absent
 - the DataFrame shape matches the storage class
 - non-empty outputs have first index level `time_index` with dtype `datetime64[ns, UTC]`
 - the first validation run uses explicit `hash_namespace(...)` when it touches a shared backend

{mainsequence-4.3.19 → mainsequence-4.3.22}/agent_scaffold/skills/data_publishing/meta_tables/SKILL.md

@@ -387,6 +387,24 @@ Only use physical table names returned by registered `MetaTable` objects when co
 
 Do not hardcode platform-managed physical names manually.
 
+### 7. DataNode storage deletes use the DataNode tail-delete API
+
+For `PlatformTimeIndexMetaTable` storage owned by DataNodes, do not design raw
+SQL delete operations or compiled SQL delete operations for rollback, repair, or
+stream cleanup. Route that work to the DataNode skill and use
+`TimeIndexMetaTable.delete_after_date(...)`.
+
+The DataNode delete path is:
+
+```text
+POST /orm/api/ts_manager/dynamic_table/<dynamic_table_uid>/delete_after_date/
+```
+
+Use `after_date` for global tail rollback. Use `dimension_filters` or
+`index_coordinates` for scoped deletes, including scoped full-stream deletes
+where `after_date=None`. Never allow `after_date=None` without an explicit
+dimension or coordinate scope.
+
 ## Review Rules
 
 When reviewing an existing MetaTable workflow, look for:
@@ -406,6 +424,8 @@ When reviewing an existing MetaTable workflow, look for:
 - migration work that asks users to define backend payloads, artifact rows, or SDK request objects
 - compiled SQL operations without complete table scope
 - raw SQL that hardcodes stale physical names
+- raw SQL or compiled SQL deletes against DataNode-owned
+  `PlatformTimeIndexMetaTable` storage instead of `delete_after_date(...)`
 - a table that should really be modeled as a DataNode instead
 
 ## Validation Checklist

{mainsequence-4.3.19 → mainsequence-4.3.22}/mainsequence/cli/cli.py

@@ -11450,6 +11450,66 @@ def project_sync(
     success(f"Synced: {repo_name}")
 
 
+@project.command("sync-after-commit")
+def project_sync_after_commit(
+    project_uid: str | None = typer.Argument(
+        None,
+        help="Project UID. If omitted, read MAIN_SEQUENCE_PROJECT_UID from local .env.",
+    ),
+    path: str | None = typer.Option(
+        None,
+        "--path",
+        help="Project directory used to resolve MAIN_SEQUENCE_PROJECT_UID when PROJECT_UID is omitted.",
+    ),
+    timeout: int | None = typer.Option(None, "--timeout", help="Request timeout in seconds."),
+):
+    """
+    Trigger backend post-commit project sync.
+
+    This directly calls:
+
+    POST /orm/api/pods/projects/<project_uid>/sync_project_after_commit/
+
+    Examples
+    --------
+    ```bash
+    mainsequence project sync-after-commit project-uid-123
+    mainsequence project sync-after-commit --path .
+    ```
+    """
+    _require_login()
+
+    resolved_project_uid = project_uid
+    if resolved_project_uid is None:
+        project_dir = normalize_path(path) if path else pathlib.Path.cwd()
+        if path and not project_dir.exists():
+            error(f"Folder does not exist: {project_dir}")
+            raise typer.Exit(1)
+        resolved_project_uid = _read_project_ref_from_env_file(project_dir)
+
+    if resolved_project_uid is None:
+        error(
+            "Could not determine project uid. Pass PROJECT_UID or ensure "
+            "MAIN_SEQUENCE_PROJECT_UID is present in local .env."
+        )
+        raise typer.Exit(1)
+
+    try:
+        payload = sync_project_after_commit(resolved_project_uid, timeout=timeout)
+    except ApiError as e:
+        error(f"Backend post-commit sync failed: {e}")
+        raise typer.Exit(1) from e
+
+    result = payload or {
+        "project_uid": resolved_project_uid,
+        "detail": "sync_project_after_commit triggered",
+    }
+    if _emit_json(result):
+        return
+
+    success(f"Triggered backend sync for project {resolved_project_uid}.")
+
+
 @project.command("sync_project", hidden=True)
 def project_sync_project(
     message: str = typer.Argument(..., help="Git commit message"),

{mainsequence-4.3.19 → mainsequence-4.3.22}/mainsequence/client/metatables/core.py

@@ -500,9 +500,17 @@ class MetaTableOperationScope(BasePydanticModel):
         return self
 
 
+DEFAULT_META_TABLE_OPERATION_MAX_ROWS = 10_000
+DEFAULT_META_TABLE_OPERATION_STATEMENT_TIMEOUT_MS = 60_000
+
+
 class MetaTableOperationLimits(BasePydanticModel):
-    max_rows: int | None = Field(default=None, ge=1)
-    statement_timeout_ms: int | None = Field(default=None, ge=1)
+    max_rows: int = Field(default=DEFAULT_META_TABLE_OPERATION_MAX_ROWS, ge=1)
+    offset: int = Field(default=0, ge=0)
+    statement_timeout_ms: int = Field(
+        default=DEFAULT_META_TABLE_OPERATION_STATEMENT_TIMEOUT_MS,
+        ge=1,
+    )
 
 
 class MetaTableCompiledSQLOperation(BasePydanticModel):
@@ -511,7 +519,12 @@ class MetaTableCompiledSQLOperation(BasePydanticModel):
     dialect: MetaTableCompiledSQLDialect = "postgresql"
     statement: MetaTableStatementPayload
     scope: MetaTableOperationScope
-    limits: MetaTableOperationLimits | None = None
+    limits: MetaTableOperationLimits = Field(default_factory=MetaTableOperationLimits)
+
+    @field_validator("limits", mode="before")
+    @classmethod
+    def default_limits(cls, value):
+        return {} if value is None else value
 
 
 class MetaTableRequestFields(BasePydanticModel):
@@ -1704,12 +1717,68 @@ class MetaTable(BasePydanticModel, LabelableObjectMixin, ShareableObjectMixin, B
             if isinstance(operation, MetaTableCompiledSQLOperation)
             else MetaTableCompiledSQLOperation(**operation)
         )
-        return cls._post_action(
+        response = cls._post_action(
             "execute-operation",
             payload,
             timeout=timeout,
             expected_statuses=(200,),
         )
+        if payload.operation != "select":
+            return response
+
+        rows = response.get("rows")
+        if not isinstance(rows, list):
+            return response
+
+        requested_max_rows = payload.limits.max_rows
+        if len(rows) >= requested_max_rows:
+            return response
+
+        pagination = response.get("pagination") or {}
+        has_more = bool(pagination.get("has_more"))
+        next_offset = pagination.get("next_offset")
+        if not has_more or next_offset in (None, ""):
+            return response
+
+        accumulated_rows = list(rows)
+        last_pagination = pagination
+        while (
+            has_more
+            and next_offset not in (None, "")
+            and len(accumulated_rows) < requested_max_rows
+        ):
+            remaining_rows = requested_max_rows - len(accumulated_rows)
+            next_payload = payload.model_copy(deep=True)
+            next_payload.limits.offset = int(next_offset)
+            next_payload.limits.max_rows = remaining_rows
+            page_response = cls._post_action(
+                "execute-operation",
+                next_payload,
+                timeout=timeout,
+                expected_statuses=(200,),
+            )
+            page_rows = page_response.get("rows") or []
+            if not isinstance(page_rows, list) or not page_rows:
+                last_pagination = page_response.get("pagination") or {}
+                break
+
+            accumulated_rows.extend(page_rows[:remaining_rows])
+            last_pagination = page_response.get("pagination") or {}
+            has_more = bool(last_pagination.get("has_more"))
+            next_offset = last_pagination.get("next_offset")
+
+        response["rows"] = accumulated_rows
+        response["row_count"] = len(accumulated_rows)
+        response["max_rows"] = requested_max_rows
+        response["truncated"] = bool(last_pagination.get("has_more"))
+        response["pagination"] = {
+            "limit": requested_max_rows,
+            "offset": payload.limits.offset,
+            "returned_count": len(accumulated_rows),
+            "has_more": bool(last_pagination.get("has_more")),
+            "next_offset": last_pagination.get("next_offset"),
+        }
+        return response
 
 
 # Global executor (or you could define one on your class)

{mainsequence-4.3.19 → mainsequence-4.3.22}/mainsequence/client/models_foundry.py

@@ -183,6 +183,16 @@ class Project(LabelableObjectMixin, ShareableObjectMixin, BasePydanticModel, Bas
         examples=["git@github.com:mainsequence/data-pipeline.git"],
         json_schema_extra={"label": "Git SSH URL"},
     )
+    latest_git_version: str = Field(
+        "",
+        title="Latest Git Version",
+        description=(
+            "Normalized highest valid version extracted from repository tags for the "
+            "project's configured branch. Empty when the backend has not found a valid version."
+        ),
+        examples=["1.2.3"],
+        json_schema_extra={"label": "Latest Git Version"},
+    )
     created_by: str | int | dict[str, Any] | None = Field(
         None,
         title="Created By",

{mainsequence-4.3.19 → mainsequence-4.3.22/mainsequence.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.3.19
+Version: 4.3.22
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement

{mainsequence-4.3.19 → mainsequence-4.3.22}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mainsequence"
-version = "4.3.19"
+version = "4.3.22"
 description = "Main Sequence SDK "
 readme = "README.md"
 requires-python = ">=3.11"

{mainsequence-4.3.19 → mainsequence-4.3.22}/tests/test_cli.py

@@ -10294,6 +10294,59 @@ def test_project_sync_triggers_backend_sync_after_push(cli_mod, runner, monkeypa
     assert "Triggered backend sync for project project-uid-123." in result.output
 
 
+def test_project_sync_after_commit_uses_project_uid(cli_mod, runner, monkeypatch):
+    captured = {}
+
+    monkeypatch.setattr(cli_mod, "_require_login", lambda: {"username": "u"})
+    monkeypatch.setattr(
+        cli_mod,
+        "sync_project_after_commit",
+        lambda project_uid, timeout=None: captured.update(
+            project_uid=project_uid,
+            timeout=timeout,
+        )
+        or {"uid": project_uid},
+    )
+
+    result = runner.invoke(
+        cli_mod.app,
+        ["project", "sync-after-commit", "project-uid-123", "--timeout", "30"],
+    )
+
+    assert result.exit_code == 0
+    assert captured == {"project_uid": "project-uid-123", "timeout": 30}
+    assert "Triggered backend sync for project project-uid-123." in result.output
+
+
+def test_project_sync_after_commit_reads_project_uid_from_env_file(
+    cli_mod, runner, monkeypatch, tmp_path
+):
+    target = tmp_path / "project"
+    target.mkdir()
+    (target / ".env").write_text("MAIN_SEQUENCE_PROJECT_UID=project-uid-123\n", encoding="utf-8")
+
+    monkeypatch.setattr(cli_mod, "_require_login", lambda: {"username": "u"})
+    monkeypatch.setattr(
+        cli_mod,
+        "sync_project_after_commit",
+        lambda project_uid, timeout=None: {
+            "project_uid": project_uid,
+            "status": "queued",
+        },
+    )
+
+    result = runner.invoke(
+        cli_mod.app,
+        ["project", "sync-after-commit", "--path", str(target), "--json"],
+    )
+
+    assert result.exit_code == 0
+    assert json.loads(result.output) == {
+        "project_uid": "project-uid-123",
+        "status": "queued",
+    }
+
+
 def test_project_sync_project(cli_mod, runner, monkeypatch, tmp_path):
     target = tmp_path / "project"
     target.mkdir(parents=True, exist_ok=True)

{mainsequence-4.3.19 → mainsequence-4.3.22}/tests/test_meta_tables_client_models.py

@@ -410,6 +410,118 @@ def test_meta_table_execute_operation_serializes_scope_uid(monkeypatch):
     assert captured["payload"]["json"]["statement"]["parameters"] == {
         "symbol_1": "%BTC%",
     }
+    assert captured["payload"]["json"]["limits"]["offset"] == 0
+
+
+def test_meta_table_execute_operation_fetches_requested_rows_with_backend_pagination(
+    monkeypatch,
+):
+    calls = []
+
+    def fake_make_request(**kwargs):
+        payload = kwargs["payload"]["json"]
+        calls.append(payload)
+        offset = payload["limits"]["offset"]
+        if offset == 0:
+            return _Response(
+                {
+                    "ok": True,
+                    "operation": "select",
+                    "dialect": "postgresql",
+                    "row_count": 2,
+                    "rows": [{"value": 1}, {"value": 2}],
+                    "truncated": True,
+                    "max_rows": 2,
+                    "pagination": {
+                        "limit": 2,
+                        "offset": 0,
+                        "returned_count": 2,
+                        "has_more": True,
+                        "next_offset": 2,
+                    },
+                }
+            )
+        if offset == 2:
+            return _Response(
+                {
+                    "ok": True,
+                    "operation": "select",
+                    "dialect": "postgresql",
+                    "row_count": 2,
+                    "rows": [{"value": 3}, {"value": 4}],
+                    "truncated": True,
+                    "max_rows": 2,
+                    "pagination": {
+                        "limit": 2,
+                        "offset": 2,
+                        "returned_count": 2,
+                        "has_more": True,
+                        "next_offset": 4,
+                    },
+                }
+            )
+        return _Response(
+            {
+                "ok": True,
+                "operation": "select",
+                "dialect": "postgresql",
+                "row_count": 1,
+                "rows": [{"value": 5}],
+                "truncated": False,
+                "max_rows": 1,
+                "pagination": {
+                    "limit": 1,
+                    "offset": 4,
+                    "returned_count": 1,
+                    "has_more": False,
+                    "next_offset": None,
+                },
+            }
+        )
+
+    monkeypatch.setattr(meta_table_models, "make_request", fake_make_request)
+    monkeypatch.setattr(
+        meta_table_models.MetaTable,
+        "build_session",
+        classmethod(lambda cls: SimpleNamespace(headers={})),
+    )
+
+    result = meta_table_models.MetaTable.execute_operation(
+        {
+            "operation": "select",
+            "statement": {
+                "sql": "SELECT value FROM public.asset ORDER BY value",
+                "parameters": {},
+            },
+            "scope": {
+                "data_source_uid": "dddddddd-dddd-4ddd-8ddd-dddddddddddd",
+                "tables": [
+                    {
+                        "meta_table_uid": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
+                        "alias": "asset",
+                    }
+                ],
+            },
+            "limits": {
+                "max_rows": 5,
+                "statement_timeout_ms": 15000,
+            },
+        }
+    )
+
+    assert [row["value"] for row in result["rows"]] == [1, 2, 3, 4, 5]
+    assert result["row_count"] == 5
+    assert result["max_rows"] == 5
+    assert result["truncated"] is False
+    assert result["pagination"] == {
+        "limit": 5,
+        "offset": 0,
+        "returned_count": 5,
+        "has_more": False,
+        "next_offset": None,
+    }
+    assert [call["limits"]["offset"] for call in calls] == [0, 2, 4]
+    assert [call["limits"]["max_rows"] for call in calls] == [5, 3, 1]
 
 
 def test_dynamic_table_data_source_issue_migration_connection_posts_scope(monkeypatch):
@@ -728,6 +840,31 @@ def test_compiled_sql_v1_protocol_is_validated_by_pydantic():
         )
 
 
+def test_compiled_sql_v1_operation_uses_backend_limit_defaults():
+    operation = build_operation(
+        operation="select",
+        sql="SELECT asset.symbol FROM public.asset AS asset",
+        scope={
+            "dataSourceUid": "dddddddd-dddd-4ddd-8ddd-dddddddddddd",
+            "tables": [
+                {
+                    "metaTableUid": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
+                    "alias": "asset",
+                }
+            ]
+        },
+    )
+
+    assert operation.limits.max_rows == 10_000
+    assert operation.limits.offset == 0
+    assert operation.limits.statement_timeout_ms == 60_000
+    assert meta_table_models._payload_json(operation)["limits"] == {
+        "max_rows": 10_000,
+        "offset": 0,
+        "statement_timeout_ms": 60_000,
+    }
+
+
 def test_compiled_sql_v1_scope_resolves_session_data_source(monkeypatch):
     monkeypatch.setattr(
         meta_table_models,
@@ -774,7 +911,7 @@ def test_compiled_sql_v1_serializes_typed_temporal_parameters():
                     "metaTableUid": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
                     "alias": "asset",
                 }
-            ]
+            ],
         },
     )