mainsequence 4.1.9__tar.gz → 4.1.11__tar.gz

{mainsequence-4.1.9 → mainsequence-4.1.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.1.9
+Version: 4.1.11
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement

{mainsequence-4.1.9 → mainsequence-4.1.11}/agent_scaffold/skills/data_publishing/data_nodes/SKILL.md

@@ -10,13 +10,13 @@ description: Use this skill when the task is about producing, changing, validati
 Use this skill when the task changes a DataNode producer.
 
 A DataNode is an update process. It is not the canonical storage model. Storage
-is defined by a registered `PlatformTimeIndexMetaData` SQLAlchemy model.
+is defined by a `PlatformTimeIndexMetaData` SQLAlchemy model.
 
 Canonical workflow:
 
 1. Define a `PlatformTimeIndexMetaData` storage class.
-2. Register that storage class before constructing the DataNode.
-3. Construct the DataNode with `config=...` and `storage_table=StorageClass`.
+2. Construct the DataNode with `config=...` and `storage_table=StorageClass`.
+3. Let the SDK register the output storage class automatically when needed.
 4. Return a DataFrame from `update()` that matches the storage class contract.
 
 ## This Skill Can Do
@@ -41,12 +41,13 @@ Canonical workflow:
 This skill must not claim ownership of:
 
 - generic MetaTable registration or governed operation semantics
-- storage creation inside `DataNode` or `PersistManager`
+- storage registration internals beyond the SDK lifecycle call
 - HTTP route design or FastAPI response contracts
 - workspace/widget layout payloads
 - job creation, scheduling, image pinning, or release creation
 - RBAC or sharing policy
 - domain strategy semantics
+- ad hoc storage creation or registration inside `update()`
 
 If the task depends on one of those areas, route it explicitly instead of guessing.
 
@@ -75,7 +76,7 @@ If the task depends on one of those areas, route it explicitly instead of guessi
 Before changing code, collect or infer:
 
 - dataset meaning
-- registered `PlatformTimeIndexMetaData` storage class or the class to create
+- `PlatformTimeIndexMetaData` output storage class or the class to create
 - expected time index and identity index shape
 - expected columns and dtypes from the storage class
 - upstream dependencies
@@ -91,7 +92,7 @@ For every non-trivial DataNode task, make these decisions explicitly:
 
 1. Is this a new dataset or the same dataset?
 2. Is this change storage-contract work or update-process work?
-3. Is the storage class registered through its MetaTable path, or should the MetaTable skill handle it?
+3. Is the storage class contract correct, or should the MetaTable skill handle it?
 4. Is the node single-index or MultiIndex?
 5. Does the first validation run happen under an explicit `hash_namespace(...)`?
 
@@ -112,6 +113,11 @@ Every storage class must include `__metatable_description__`. The description
 should explain the table's intention, row grain, and downstream use, not only
 list columns or schema mechanics.
 
+Every `mapped_column(...)` in a storage class must include
+`info={"label": ..., "description": ...}`. The column description must explain
+what the value means for this dataset and how downstream users should interpret
+it; do not merely repeat the column name or dtype.
+
 Use `__metatable_extra_hash_components__` on storage classes when distinct
 DataNode storage tables could otherwise have the same storage-relevant shape.
 For example, two one-index daily tables with one float column need a stable
@@ -153,18 +159,38 @@ class PricesTable(PlatformTimeIndexMetaData, Base):
     time_index: Mapped[datetime.datetime] = mapped_column(
         DateTime(timezone=True),
         nullable=False,
+        info={
+            "label": "Time Index",
+            "description": "UTC timestamp for the close-price observation.",
+        },
+    )
+    unique_identifier: Mapped[str] = mapped_column(
+        nullable=False,
+        info={
+            "label": "Unique Identifier",
+            "description": "Stable asset identifier for the observed price.",
+        },
+    )
+    close: Mapped[float] = mapped_column(
+        Float,
+        nullable=False,
+        info={
+            "label": "Close",
+            "description": "Daily close price used by portfolio and risk analytics.",
+        },
     )
-    unique_identifier: Mapped[str] = mapped_column(nullable=False)
-    close: Mapped[float] = mapped_column(Float, nullable=False)
 ```
 
-Register storage before constructing the DataNode:
+Storage registration is inferred from the class metadata and active Main
+Sequence project/session. The DataNode constructor ensures the output
+`storage_table` is registered when needed. You may still call it explicitly
+during bootstrap when code needs the returned metadata object:
 
 ```python
-PricesTable.register(data_source_uid=data_source_uid)
+PricesTable.register()
 ```
 
-`PlatformTimeIndexMetaData.register(...)` is the storage lifecycle path. Treat it
+`PlatformTimeIndexMetaData.register()` is the storage lifecycle path. Treat it
 as an idempotent get-or-create operation: the platform returns the registered
 metadata and UID, and the SDK records that metadata on the class. Do not manually
 attach an existing UID, reconstruct a generic `MetaTable`, or use manual bind
@@ -175,18 +201,21 @@ helpers as an authoring step.
 The DataNode constructor should accept:
 
 - a `DataNodeConfiguration`
-- a registered `storage_table: type[PlatformTimeIndexMetaData]`
+- a `storage_table: type[PlatformTimeIndexMetaData]`
 - optional `hash_namespace`
 
 The constructor `storage_table` is the output storage contract. Keep it out of
-`DataNodeConfiguration`.
+`DataNodeConfiguration`. Do not pre-register this output storage class just to
+construct the node; `DataNode` and `PersistManager` call the SDK registration
+lifecycle automatically when the class is not yet bound.
 
 If the DataNode needs to select another DataNode's storage table as a
 dependency, put that dependency storage reference in the config as
 `type[PlatformTimeIndexMetaData]`. Do not add an extra constructor argument for
 dependency storage tables. Config values of this type are hashed by the bound
-`TimeIndexMetaData.uid` from `StorageClass.__time_index_metadata__`, so the class
-must be registered before DataNode construction.
+`TimeIndexMetaData.uid` from `StorageClass.__time_index_metadata__`; if the
+class is not yet bound, the config serializer calls `StorageClass.register()`
+before reading the UID.
 
 Do not accept `test_node`. It has been removed. Use explicit
 `hash_namespace(...)` or `hash_namespace="..."`.
@@ -359,6 +388,10 @@ DataNode storage table needs a platform-managed FK, use
 `ForeignKey(Target.__table__.c.uid)`, table fullnames, or explicit target UID
 maps in DataNode examples.
 
+Do not ask users to name these foreign keys. `MetaTableForeignKey(...)` derives
+a stable contract name when `name` is omitted; `name=...` is only for deliberate
+overrides.
+
 Registration of the storage class follows the MetaTable lifecycle:
 `register()` recursively registers unresolved FK target model classes, uses the
 local process registry keyed by `storage_hash`, and writes the target
@@ -379,11 +412,13 @@ When reviewing an existing DataNode, look for:
 
 - output storage contract hidden in `DataNodeConfiguration`
 - missing `__metatable_description__` on the storage table
+- storage columns without `mapped_column(info={"label": ..., "description": ...})`
 - dependency storage table passed as an ad hoc constructor argument
 - schema or published table metadata hidden in DataNode configuration
 - `test_node=True`
 - missing explicit `storage_table`
-- accidental storage registration inside the DataNode
+- manual pre-registration of the output `storage_table` when no returned
+  metadata object is needed
 - wrong split between hashed config fields and non-config class/runtime values
 - misuse of `hash_namespace`
 - non-incremental `update()` behavior
@@ -397,10 +432,12 @@ When reviewing an existing DataNode, look for:
 Do not claim success until you have checked:
 
 - the relevant docs were read first
-- storage is a registered `PlatformTimeIndexMetaData` class
+- output storage is a `PlatformTimeIndexMetaData` class that the SDK can register
 - storage has an intention-rich `__metatable_description__`
+- every storage column has an intention-rich `info.description`
 - the DataNode constructor requires `storage_table`
-- dependency storage-table references live in config and are registered
+- dependency storage-table references live in config and rely on SDK
+  registration during config serialization
 - config fields are updater-scoped by default
 - no removed hash metadata markers remain
 - no `test_node` usage remains

{mainsequence-4.1.9 → mainsequence-4.1.11}/agent_scaffold/skills/data_publishing/meta_tables/SKILL.md

@@ -102,6 +102,9 @@ Schema must come from SQLAlchemy table metadata, usually `__table_args__ = {"sch
 Always declare `__metatable_description__` on the model. The description must
 explain the table's business intention, row grain, and expected use, not only
 the schema. Column-level descriptions stay in `mapped_column(info={...})`.
+Every mapped column must include `info={"label": ..., "description": ...}`.
+The column description must explain what the value means in this table and how
+it is used, not just restate the column name or dtype.
 
 Use `__metatable_extra_hash_components__` when two backend-managed tables could
 otherwise produce the same storage hash because their storage-relevant shape is
@@ -124,13 +127,32 @@ class Account(PlatformManagedMetaTable, Base):
         "Customer account master records used to scope balances, holdings, and "
         "account-level limits."
     )
+    __metatable_labels__ = ["sdk-example"]
+
+    uid: Mapped[uuid.UUID] = mapped_column(
+        Uuid,
+        primary_key=True,
+        info={
+            "label": "Account UID",
+            "description": "Stable account identifier referenced by dependent tables.",
+        },
+    )
+    name: Mapped[str] = mapped_column(
+        String(255),
+        nullable=False,
+        info={
+            "label": "Name",
+            "description": "Display name used to identify the account in examples.",
+        },
+    )
 
 
-account_meta_table = Account.register(labels=["sdk-example"])
+account_meta_table = Account.register()
 ```
 
-Pass `description=...` only when the call intentionally overrides the class
-default.
+Registration metadata belongs on the class. Do not pass description, labels,
+provisioning, data-source UID, hash namespace, time-index fields, or storage
+layout into `register()`.
 
 For platform-managed registration, the data source is resolved from the active Main Sequence project/session, the same way DataNode does. Do not require or thread a `data_source_uid` through normal platform-managed example code.
 
@@ -160,6 +182,11 @@ maps in the platform-managed path. Registration is the lifecycle path:
 returned `MetaTable` in a local process registry keyed by `storage_hash`, and
 uses the target `MetaTable.uid` in the child FK contract.
 
+Do not require users to provide foreign-key names. `MetaTableForeignKey(...)`
+accepts `name=...` only as an override; when omitted, the SDK derives a stable
+PostgreSQL-safe contract name from the child table and source column after the
+column is attached to the SQLAlchemy table.
+
 Use this pattern:
 
 ```python
@@ -176,7 +203,7 @@ both the schema and the table's intention.
 Example registration order:
 
 ```python
-asset_meta_table = Asset.register(...)
+asset_meta_table = Asset.register()
 ```
 
 The child registration registers `Account` first if it has not already been
@@ -216,11 +243,13 @@ When reviewing an existing MetaTable workflow, look for:
 
 - missing namespace or identifier
 - missing `__metatable_description__`, or a description that only repeats column names instead of table intention
+- mapped columns without `info.label` and `info.description`
 - backend-managed models that do not inherit `PlatformManagedMetaTable`
 - backend-managed examples that use namespace environment variables instead of a plain `sdk-examples` namespace
 - duplicate schema sources outside SQLAlchemy table metadata
 - external tables registered with unstable physical names
-- platform-managed child tables that are registered before parent tables
+- platform-managed examples that manually sequence parent registration instead
+  of relying on `MetaTableForeignKey(...)` recursive registration
 - external child registrations that do not map foreign-key targets to registered parent `MetaTable.uid` values
 - compiled SQL operations without complete table scope
 - raw SQL that hardcodes stale physical names
@@ -232,6 +261,7 @@ Do not claim success until you have checked:
 
 - the table contract matches the intended row contract
 - the table has an intention-rich `__metatable_description__`
+- every mapped column has an intention-rich `info.description`
 - indexes are intentional
 - foreign keys resolve to the correct dependency targets
 - management mode is correct
@@ -242,7 +272,8 @@ Do not claim success until you have checked:
 For related tables, also check:
 
 - aliases are readable
-- platform-managed parent tables are registered before child tables
+- platform-managed child registration recursively resolves parent
+  `MetaTableForeignKey(...)` targets
 - external child registration requests map FK targets to the registered parent UIDs
 - query results still match the expected response contract
 

{mainsequence-4.1.9 → mainsequence-4.1.11}/mainsequence/meta_tables/data_nodes/build_operations.py

@@ -94,10 +94,15 @@ def _(value: type[Any]) -> Any:
 
     time_index_metadata = value.get_time_index_metadata()
     uid = getattr(time_index_metadata, "uid", None)
+    if uid in (None, ""):
+        value.register()
+        time_index_metadata = value.get_time_index_metadata()
+        uid = getattr(time_index_metadata, "uid", None)
+
     if uid in (None, ""):
         raise ValueError(
-            "PlatformTimeIndexMetaData config values must be registered "
-            "before they can be hashed."
+            "PlatformTimeIndexMetaData config value register() did not bind "
+            "TimeIndexMetaData metadata before hashing."
         )
     return {
         "__type__": "platform_time_index_metadata",

{mainsequence-4.1.9 → mainsequence-4.1.11}/mainsequence/meta_tables/data_nodes/data_nodes.py

@@ -29,7 +29,11 @@ from mainsequence.client.utils import META_TABLES_CONSTANTS as CONSTANTS
 from mainsequence.instrumentation import tracer
 from mainsequence.logconf import logger
 from mainsequence.meta_tables import PlatformTimeIndexMetaData
-from mainsequence.meta_tables.data_nodes.persist_managers import APIPersistManager, PersistManager
+from mainsequence.meta_tables.data_nodes.persist_managers import (
+    APIPersistManager,
+    PersistManager,
+    ensure_registered_storage_table,
+)
 
 from .models import BaseConfiguration
 from .namespacing import current_hash_namespace
@@ -625,34 +629,15 @@ class DataNode(DataAccessMixin, ABC):
 
     @storage_table.setter
     def storage_table(self, value: type[PlatformTimeIndexMetaData]) -> None:
-        if value is None:
-            raise TypeError(
-                "DataNode storage_table is required and must be a "
-                "PlatformTimeIndexMetaData model class."
-            )
-        is_time_index_model_class = isinstance(value, type) and issubclass(
-            value, PlatformTimeIndexMetaData
-        )
-        if not is_time_index_model_class:
-            raise TypeError(
-                "DataNode storage_table must be a PlatformTimeIndexMetaData model class; "
-                f"got {type(value).__name__}."
-            )
-        if value.get_time_index_metadata() is None:
-            raise ValueError(
-                "DataNode storage_table must be registered before construction."
-            )
-        if value.get_meta_table_uid() in (None, ""):
-            raise ValueError("DataNode storage_table must provide a MetaTable UID.")
-        if value.get_data_source_uid() in (None, ""):
-            raise ValueError("DataNode storage_table must provide a data-source UID.")
-        self._storage_table = value
+        self._storage_table = ensure_registered_storage_table(value, context="DataNode")
 
     @property
     def storage_metadata(self) -> Any:
         storage_metadata = self.storage_table.get_time_index_metadata()
         if storage_metadata is None:
-            raise ValueError("DataNode storage_table must be registered before use.")
+            raise ValueError(
+                "DataNode storage_table registration metadata is unavailable after register()."
+            )
         return storage_metadata
 
     def _initialize_configuration(self, init_kwargs: dict) -> None:

{mainsequence-4.1.9 → mainsequence-4.1.11}/mainsequence/meta_tables/data_nodes/persist_managers.py

@@ -77,6 +77,40 @@ def get_data_node_source_code_git_hash(DataNodeClass: type[Any]) -> str:
     return hash_object.hexdigest()
 
 
+def ensure_registered_storage_table(
+    storage_table: type[PlatformTimeIndexMetaData],
+    *,
+    context: str,
+) -> type[PlatformTimeIndexMetaData]:
+    if storage_table is None:
+        raise TypeError(
+            f"{context} storage_table is required and must be a "
+            "PlatformTimeIndexMetaData model class."
+        )
+    if not isinstance(storage_table, type) or not issubclass(
+        storage_table,
+        PlatformTimeIndexMetaData,
+    ):
+        raise TypeError(
+            f"{context} storage_table must be a PlatformTimeIndexMetaData "
+            f"model class; got {type(storage_table).__name__}."
+        )
+
+    if storage_table.get_time_index_metadata() is None:
+        storage_table.register()
+
+    storage_metadata = storage_table.get_time_index_metadata()
+    if storage_metadata is None:
+        raise ValueError(
+            f"{context} storage_table.register() did not bind TimeIndexMetaData metadata."
+        )
+    if storage_table.get_meta_table_uid() in (None, ""):
+        raise ValueError(f"{context} storage_table must provide a MetaTable UID.")
+    if storage_table.get_data_source_uid() in (None, ""):
+        raise ValueError(f"{context} storage_table must provide a data-source UID.")
+    return storage_table
+
+
 class BasePersistManager:
     UPDATE_CLASS: ClassVar[type[Any] | None] = None
     UPDATE_DETAILS_CLASS: ClassVar[type[Any] | None] = None
@@ -114,36 +148,15 @@ class BasePersistManager:
     def _validate_storage_table(
         storage_table: type[PlatformTimeIndexMetaData],
     ) -> type[PlatformTimeIndexMetaData]:
-        if storage_table is None:
-            raise TypeError(
-                "PersistManager storage_table is required and must be a "
-                "PlatformTimeIndexMetaData model class."
-            )
-        if not isinstance(storage_table, type) or not issubclass(
-            storage_table,
-            PlatformTimeIndexMetaData,
-        ):
-            raise TypeError(
-                "PersistManager storage_table must be a PlatformTimeIndexMetaData "
-                f"model class; got {type(storage_table).__name__}."
-            )
-        storage_metadata = storage_table.get_time_index_metadata()
-        if storage_metadata is None:
-            raise ValueError(
-                "PersistManager storage_table must be registered before "
-                "constructing the DataNode."
-            )
-        if storage_table.get_meta_table_uid() in (None, ""):
-            raise ValueError("PersistManager storage_table must provide a MetaTable UID.")
-        if storage_table.get_data_source_uid() in (None, ""):
-            raise ValueError("PersistManager storage_table must provide a data-source UID.")
-        return storage_table
+        return ensure_registered_storage_table(storage_table, context="PersistManager")
 
     @property
     def storage_metadata(self) -> Any:
         storage_metadata = self.storage_table.get_time_index_metadata()
         if storage_metadata is None:
-            raise ValueError("PersistManager storage_table must be registered before use.")
+            raise ValueError(
+                "PersistManager storage_table registration metadata is unavailable after register()."
+            )
         return storage_metadata
 
     @property