mainsequence 4.0.3__tar.gz → 4.0.5__tar.gz

{mainsequence-4.0.3 → mainsequence-4.0.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.0.3
+Version: 4.0.5
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement

{mainsequence-4.0.3 → mainsequence-4.0.5}/agent_scaffold/skills/data_publishing/meta_tables/SKILL.md

@@ -15,7 +15,8 @@ This skill is for schema-driven application tables registered through TS Manager
 
 - define SQLAlchemy/Core or ORM table models for `MetaTable` registration
 - choose `platform_managed` or `external_registered` management mode
-- build registration requests from resolved SQLAlchemy metadata
+- register platform-managed tables through the model class API
+- build registration requests from resolved SQLAlchemy metadata when inspection is useful
 - define indexes and foreign keys in the table contract
 - design governed compiled SQL read and write operations
 - review table contracts for physical-name, namespace, and identifier issues
@@ -67,7 +68,7 @@ Before changing code, collect or infer:
 - expected read patterns
 - expected mutation patterns
 - whether TS Manager should create the physical table
-- the target `DynamicTableDataSource` UID
+- for `external_registered`, the target `DynamicTableDataSource` UID
 
 If ownership of the physical table lifecycle is unclear, stop before choosing a management mode.
 
@@ -92,15 +93,67 @@ Do not hand-build contract fragments when the SQLAlchemy helper can derive them.
 
 ### 2. Use storage-hash physical names for backend-managed tables
 
-For `platform_managed`, use `metatable_tablename(...)` as the SQLAlchemy `__tablename__`.
+For `platform_managed`, inherit from `PlatformManagedMetaTable`.
 
-The backend expects that deterministic physical name to match the registration storage hash.
+The mixin derives the SQLAlchemy physical table name from storage-relevant configuration and table shape. Do not hand-write `__tablename__` for normal backend-managed tables.
+
+Schema must come from SQLAlchemy table metadata, usually `__table_args__ = {"schema": "public"}` or the tuple form ending in `{"schema": ...}`. Do not add a separate MetaTable-specific schema attribute.
+
+Register through the class API:
+
+```python
+account_meta_table = Account.register(
+    description="Example account table.",
+    labels=["sdk-example"],
+)
+```
+
+For platform-managed registration, the data source is resolved from the active Main Sequence project/session, the same way DataNode does. Do not require or thread a `data_source_uid` through normal platform-managed example code.
+
+Only call `build_registration_request()` when the task explicitly needs to inspect or validate the payload before registration.
+
+For SDK examples, use a plain namespace constant:
+
+```python
+NAMESPACE = "sdk-examples"
+```
+
+Do not add an environment variable for namespace in examples.
+
+Do not add generic labels such as `"meta-table"` or `"platform-managed"` to examples. Keep labels specific to the example or domain.
+
+Do not add a `MAINSEQUENCE_META_TABLE_REGISTER` toggle in registration examples. Registration examples should register directly.
 
 ### 3. Register parent tables before child tables
 
 Foreign-key contracts reference the target `MetaTable` UID.
 
-Register parent tables first, then pass their UIDs when building child table registration requests.
+For `PlatformManagedMetaTable`, register parent tables first and then register child tables normally. The SDK inspects SQLAlchemy foreign-key constraints and resolves each target `MetaTable` by looking up the already registered table in the same data source, schema, and physical table name.
+
+Example registration order:
+
+```python
+account_meta_table = Account.register(...)
+asset_meta_table = Asset.register(...)
+```
+
+The child registration will fail if the parent table has not already been registered, because there is no target `MetaTable.uid` for the backend FK contract.
+
+Do not pass `target_meta_tables` or `target_meta_table_uid_by_fullname` in the normal platform-managed path. Use explicit FK target mappings only for edge cases where automatic lookup is ambiguous or impossible.
+
+For `external_registered`, there is no platform-managed parent lookup through the model class. Register the parent first, then build the child registration request with the parent UID mapped by target table fullname:
+
+```python
+account_meta_table = MetaTable.register(account_request)
+asset_request = external_registered_registration_request_from_sqlalchemy_model(
+    Asset,
+    data_source_uid=data_source_uid,
+    target_meta_table_uid_by_fullname={
+        Account.__table__.fullname: account_meta_table.uid,
+    },
+)
+asset_meta_table = MetaTable.register(asset_request)
+```
 
 ### 4. Governed operations declare scope
 
@@ -119,9 +172,12 @@ Do not hardcode platform-managed physical names manually.
 When reviewing an existing MetaTable workflow, look for:
 
 - missing namespace or identifier
-- backend-managed models that do not use `metatable_tablename(...)`
+- backend-managed models that do not inherit `PlatformManagedMetaTable`
+- backend-managed examples that use namespace environment variables instead of a plain `sdk-examples` namespace
+- duplicate schema sources outside SQLAlchemy table metadata
 - external tables registered with unstable physical names
-- foreign keys that do not reference target MetaTable UIDs
+- platform-managed child tables that are registered before parent tables
+- external child registrations that do not map foreign-key targets to registered parent `MetaTable.uid` values
 - compiled SQL operations without complete table scope
 - raw SQL that hardcodes stale physical names
 - a table that should really be modeled as a DataNode instead
@@ -132,7 +188,7 @@ Do not claim success until you have checked:
 
 - the table contract matches the intended row contract
 - indexes are intentional
-- foreign keys point to the correct dependency targets
+- foreign keys resolve to the correct dependency targets
 - management mode is correct
 - backend-managed physical names match the storage hash
 - registration returns a `MetaTable.uid`
@@ -141,13 +197,14 @@ Do not claim success until you have checked:
 For related tables, also check:
 
 - aliases are readable
-- parent table UIDs are passed into child contracts
+- platform-managed parent tables are registered before child tables
+- external child registration requests map FK targets to the registered parent UIDs
 - query results still match the expected response contract
 
 ## This Skill Must Stop And Escalate When
 
 - physical table lifecycle ownership is unclear
-- the target data source is unknown
+- the target data source is unknown for an `external_registered` workflow
 - the task really requires a time-series published table
 - the workflow requires direct database credentials outside TS Manager governance
 - the task is actually an API or orchestration problem

{mainsequence-4.0.3 → mainsequence-4.0.5}/mainsequence/client/models_tdag.py

@@ -28,7 +28,6 @@ from mainsequence.logconf import logger
 from . import exceptions
 from .base import BaseObjectOrm, BasePydanticModel, LabelableObjectMixin, ShareableObjectMixin
 from .data_sources_interfaces import get_duckdb_interface_class
-from .data_sources_interfaces import timescale as TimeScaleInterface
 from .exceptions import raise_for_response
 from .utils import (
     MAINSEQUENCE_ENDPOINT,
@@ -44,7 +43,6 @@ from .utils import (
     make_request,
     serialize_to_json,
     session,
-    set_types_in_table,
 )
 
 _default_data_source = None  # Module-level cache
@@ -350,7 +348,13 @@ class DataNodeUpdate(TableUpdateNode, BaseObjectOrm):
         url = cls.get_object_url() + "/get_or_create/"
         kwargs = serialize_to_json(kwargs)
         pod_project = _require_local_pod_project("DataNodeUpdate.get_or_create")
-        kwargs["current_project_id"] = pod_project.id
+        project_uid = str(getattr(pod_project, "uid", "") or "").strip()
+        if not project_uid:
+            raise RuntimeError(
+                "DataNodeUpdate.get_or_create requires a local pod project uid, "
+                "but the resolved project does not expose one."
+            )
+        kwargs["current_project_uid"] = project_uid
         payload = {"json": kwargs}
         s = cls.build_session()
         r = make_request(s=s, loaders=cls.LOADERS, r_type="POST", url=url, payload=payload)
@@ -3150,11 +3154,23 @@ class UpdateBatchResponse(BaseModel, Generic[UpdateT, UpdateDetailsT, SourceTabl
 
 
 class DataSource(BasePydanticModel, BaseObjectOrm):
+    uid: str | None = Field(
+        None,
+        description="Public uid of the data source.",
+    )
+    data_source_uid: str | None = Field(
+        None,
+        description="Compatibility alias for the public data source uid.",
+    )
     id: int | None = Field(None, description="The unique identifier of the Local Disk Source Lake")
     display_name: str
     organization: int | None = Field(
         None, description="The unique identifier of the Local Disk Source Lake"
     )
+    organization_uid: str | None = Field(
+        None,
+        description="Public uid of the owning organization.",
+    )
     class_type: str
     status: str
     extra_arguments: dict | None = None
@@ -3307,7 +3323,14 @@ class DataSource(BasePydanticModel, BaseObjectOrm):
 
 
 class DynamicTableDataSource(BasePydanticModel, BaseObjectOrm):
-    id: int
+    uid: str | None = Field(
+        None,
+        description="Public uid of the dynamic table data source.",
+    )
+    id: int | None = Field(
+        None,
+        description="Legacy numeric identifier of the dynamic table data source.",
+    )
     related_resource: DataSource
     related_resource_class_type: str
 
@@ -3345,23 +3368,8 @@ class DynamicTableDataSource(BasePydanticModel, BaseObjectOrm):
             raise Exception(f"Error in request {r.text}")
         return cls(**r.json())
 
-    def has_direct_postgres_connection(self):
-        return self.related_resource.class_type == "direct"
-
     def get_data_by_time_index(self, *args, **kwargs):
-        if self.has_direct_postgres_connection():
-            stc = kwargs["data_node_update"].data_node_storage.sourcetableconfiguration
-
-            df = TimeScaleInterface.direct_data_from_db(
-                *args,
-                connection_uri=self.related_resource.get_connection_uri(),
-
-                **kwargs,
-            )
-            df = set_types_in_table(df, stc.column_dtypes_map)
-            return df
-        else:
-            return self.related_resource.get_data_by_time_index(*args, **kwargs)
+        return self.related_resource.get_data_by_time_index(*args, **kwargs)
 
 
 
@@ -3478,14 +3486,11 @@ class Project(LabelableObjectMixin, ShareableObjectMixin, BasePydanticModel, Bas
     FILTERSET_FIELDS: ClassVar[dict[str, list[str]]] = {
         "project_name": ["in", "exact", "contains"],
         "uid": ["in", "exact"],
-        "id": ["in", "exact"],
         "labels": ["exact", "in", "contains"],
     }
     FILTER_VALUE_NORMALIZERS: ClassVar[dict[str, str]] = {
         "uid": "str",
         "uid__in": "str",
-        "id": "id",
-        "id__in": "id",
         "project_name": "str",
         "labels": "str",
         "labels__in": "str",
@@ -3498,10 +3503,10 @@ class Project(LabelableObjectMixin, ShareableObjectMixin, BasePydanticModel, Bas
         examples=["project-uid-142"],
         json_schema_extra={"label": "Project UID"},
     )
-    id: int = Field(
-        ...,
+    id: int | None = Field(
+        None,
         title="Project ID",
-        description="Unique identifier of the project.",
+        description="Legacy numeric identifier of the project.",
         examples=[142],
         json_schema_extra={"label": "Project ID"},
     )
@@ -3525,6 +3530,12 @@ class Project(LabelableObjectMixin, ShareableObjectMixin, BasePydanticModel, Bas
         examples=["git@github.com:mainsequence/data-pipeline.git"],
         json_schema_extra={"label": "Git SSH URL"},
     )
+    created_by: str | int | dict[str, Any] | None = Field(
+        None,
+        title="Created By",
+        description="Backend-provided creator metadata for the project.",
+        json_schema_extra={"label": "Created By"},
+    )
 
     labels: list[str] = Field(
         default_factory=list,

{mainsequence-4.0.3 → mainsequence-4.0.5}/mainsequence/tdag/data_nodes/build_operations.py

@@ -245,10 +245,10 @@ def hash_signature(dictionary: dict[str, Any]) -> tuple[str, str]:
     local_ts_dict_to_hash = _strip_pydantic_hash_exclusions(parsed_dictionary, for_storage_hash=False)
     remote_ts_in_db_hash = _strip_pydantic_hash_exclusions(parsed_dictionary, for_storage_hash=True)
 
-    # Add project_id for local hash
+    # Add project_uid for local hash so local hashing follows the public project contract.
     resolution = _resolve_local_pod_project()
-    if resolution.project is not None:
-        local_ts_dict_to_hash["project_id"] = resolution.project.id
+    if resolution.project is not None and getattr(resolution.project, "uid", None):
+        local_ts_dict_to_hash["project_uid"] = resolution.project.uid
     # Encode and hash both versions
     encoded_local = json.dumps(local_ts_dict_to_hash, sort_keys=True).encode()
     encoded_remote = json.dumps(remote_ts_in_db_hash, sort_keys=True).encode()

{mainsequence-4.0.3 → mainsequence-4.0.5}/mainsequence/tdag/meta_tables/__init__.py

@@ -7,17 +7,26 @@ _LAZY_IMPORTS = {
         ".sqlalchemy_contracts",
         "DEFAULT_PLATFORM_MANAGED_PROVISIONING",
     ),
+    "PlatformManagedMetaTable": (".sqlalchemy_contracts", "PlatformManagedMetaTable"),
     "POSTGRES_IDENTIFIER_MAX_LENGTH": (".hashing", "POSTGRES_IDENTIFIER_MAX_LENGTH"),
     "build_compiled_sql_v1_operation": (
         ".compiled_sql",
         "build_compiled_sql_v1_operation",
     ),
+    "build_meta_table_configured_storage_hash": (
+        ".hashing",
+        "build_meta_table_configured_storage_hash",
+    ),
     "build_meta_table_storage_hash": (".hashing", "build_meta_table_storage_hash"),
     "compile_sqlalchemy_statement": (".compiled_sql", "compile_sqlalchemy_statement"),
     "external_registered_registration_request_from_sqlalchemy_model": (
         ".sqlalchemy_contracts",
         "external_registered_registration_request_from_sqlalchemy_model",
     ),
+    "metatable_configured_tablename": (
+        ".sqlalchemy_contracts",
+        "metatable_configured_tablename",
+    ),
     "metatable_tablename": (".sqlalchemy_contracts", "metatable_tablename"),
     "platform_managed_registration_request_from_sqlalchemy_model": (
         ".sqlalchemy_contracts",

{mainsequence-4.0.3 → mainsequence-4.0.5}/mainsequence/tdag/meta_tables/hashing.py

@@ -76,6 +76,59 @@ def build_meta_table_storage_hash(
     return storage_hash
 
 
+def build_meta_table_configured_storage_hash(
+    *,
+    namespace: str,
+    schema: str = "public",
+    table_storage_identity: Mapping[str, Any],
+    hash_namespace: str | None = None,
+    extra_hash_components: Mapping[str, Any] | None = None,
+    max_length: int = POSTGRES_IDENTIFIER_MAX_LENGTH,
+) -> str:
+    """
+    Build a PostgreSQL-safe storage hash for a configured MetaTable.
+
+    This path intentionally excludes the logical MetaTable identifier. The
+    physical table is identified by storage-relevant configuration, while the
+    identifier remains backend/display metadata.
+    """
+
+    if max_length <= _HASH_SUFFIX_LENGTH:
+        raise ValueError("max_length must leave room for the DataNode hash suffix.")
+
+    namespace = namespace.strip()
+    schema = schema.strip()
+    if not namespace:
+        raise ValueError("namespace is required to build a configured MetaTable storage hash.")
+    if not schema:
+        raise ValueError("schema is required to build a configured MetaTable storage hash.")
+
+    prefix_base = slugify_identifier(f"mt_{namespace}")
+    max_prefix_length = max_length - _HASH_SUFFIX_LENGTH
+    prefix = prefix_base[:max_prefix_length].rstrip("_") or "mt"
+
+    hash_payload: dict[str, Any] = {
+        "namespace": namespace,
+        "schema": schema,
+        "table_storage_identity": dict(table_storage_identity),
+    }
+    if hash_namespace:
+        hash_payload["hash_namespace"] = hash_namespace.strip()
+    if extra_hash_components:
+        hash_payload.update(dict(extra_hash_components))
+
+    storage_hash = _build_storage_hash_with_data_node_machinery(
+        prefix=prefix,
+        hash_payload=hash_payload,
+    )
+    if len(storage_hash) > max_length:
+        raise ValueError(
+            f"Generated MetaTable storage hash exceeds {max_length} characters: "
+            f"{storage_hash!r}."
+        )
+    return storage_hash
+
+
 def _build_storage_hash_with_data_node_machinery(
     *,
     prefix: str,
@@ -113,6 +166,7 @@ def _build_storage_hash_without_tdag_config(
 
 __all__ = [
     "POSTGRES_IDENTIFIER_MAX_LENGTH",
+    "build_meta_table_configured_storage_hash",
     "build_meta_table_storage_hash",
     "slugify_identifier",
 ]