mainsequence 4.0.3__tar.gz → 4.0.6__tar.gz

{mainsequence-4.0.3 → mainsequence-4.0.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.0.3
+Version: 4.0.6
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement

{mainsequence-4.0.3 → mainsequence-4.0.6}/agent_scaffold/skills/data_publishing/data_nodes/SKILL.md

@@ -17,12 +17,13 @@ This skill is for producer-side table engineering.
 - modify an existing `DataNode`
 - review whether a DataNode change is breaking or non-breaking
 - define or refactor `DataNodeConfiguration`
-- classify config fields into dataset meaning, updater scope, and runtime-only concerns
+- classify config fields into dataset meaning, updater scope, and hash-excluded metadata
 - implement or review:
   - `dependencies()`
   - `update()`
   - `get_asset_list()`
-  - metadata and record definitions
+  - `RecordDefinition` declarations and metadata
+  - DataNode source-table foreign key declarations to MetaTables
 - design single-index or `(time_index, unique_identifier)` MultiIndex outputs
 - define namespace-first validation strategy
 - write or review DataNode smoke tests
@@ -71,7 +72,8 @@ Before changing code, collect or infer:
 - dataset meaning
 - intended published identifier
 - expected index shape
-- expected columns and dtypes
+- expected columns and dtypes, declared as `RecordDefinition`
+- whether declared columns should reference registered MetaTables through foreign keys
 - upstream dependencies
 - whether the node is asset-indexed
 - first-run or backfill bounds
@@ -106,7 +108,8 @@ Do not change them casually.
 
 - dataset meaning belongs in table identity
 - updater scope belongs in updater identity
-- runtime-only knobs belong outside hashed identity
+- descriptive metadata belongs in `hash_excluded` fields
+- runtime knobs belong outside hashed identity
 
 Do not mix these.
 
@@ -140,12 +143,127 @@ If the node emits `(time_index, unique_identifier)`:
 - `get_asset_list()` must reflect the effective updater asset scope
 - missing assets should be resolved or registered when required by the workflow
 
-### 7. Metadata is not optional for production-quality nodes
-
-When the node is not a throwaway example, provide:
-
-- table metadata
-- column metadata or record definitions
+### 7. `RecordDefinition` is the canonical schema surface
+
+Every new or materially edited `DataNodeConfiguration` must declare output
+records with `RecordDefinition` unless there is a documented compatibility
+reason not to. Treat `records` as the canonical table schema declaration, not
+as optional UI metadata.
+
+The canonical pattern is:
+
+```python
+from pydantic import Field
+
+from mainsequence.tdag import DataNodeConfiguration, RecordDefinition
+
+
+class PricesConfig(DataNodeConfiguration):
+    records: list[RecordDefinition] = Field(
+        default_factory=lambda: [
+            RecordDefinition(
+                column_name="price",
+                dtype="float64",
+                label="Price",
+                description="Observed price.",
+            )
+        ]
+    )
+```
+
+Rules:
+
+- `column_name` and `dtype` are structural and define the persisted record contract.
+- `label` and `description` are descriptive discovery metadata and must not be treated as runtime controls.
+- The DataFrame returned by `update()` must match declared `records`.
+- Do not invent another schema object or parallel record declaration.
+- Prefer `DataNodeConfiguration.records` over overriding `get_column_metadata()` for normal nodes.
+
+### 8. Use `SourceTableForeignKey` when a DataNode references a MetaTable
+
+When a DataNode source table has a column that should reference a registered
+MetaTable, declare that relationship in `DataNodeConfiguration.foreign_keys`.
+Do this only for DataNode source-table to MetaTable relationships. Do not
+invent DataNode-to-DataNode or MetaTable-to-DataNode foreign keys.
+
+The canonical pattern is:
+
+```python
+from pydantic import Field
+
+from mainsequence.tdag import (
+    DataNodeConfiguration,
+    RecordDefinition,
+    SourceTableForeignKey,
+)
+
+
+ASSET_UID = RecordDefinition(
+    column_name="asset_uid",
+    dtype="uuid",
+    label="Asset",
+    description="Asset UID.",
+)
+
+
+class PricesConfig(DataNodeConfiguration):
+    records: list[RecordDefinition] = Field(
+        default_factory=lambda: [
+            RecordDefinition(
+                column_name="time_index",
+                dtype="datetime64[ns, UTC]",
+                label="Time",
+                description="UTC observation timestamp.",
+            ),
+            ASSET_UID,
+            RecordDefinition(
+                column_name="price",
+                dtype="float64",
+                label="Price",
+                description="Observed price.",
+            ),
+        ]
+    )
+    foreign_keys: list[SourceTableForeignKey] = Field(
+        default_factory=lambda: [
+            SourceTableForeignKey(
+                target=Asset,
+                source_columns=[ASSET_UID],
+                target_columns=[Asset.uid],
+                on_delete="restrict",
+            )
+        ]
+    )
+```
+
+Rules:
+
+- `SourceTableForeignKey` is the authoring model; do not hand-author
+  `SourceTableForeignKeyContract` in DataNode configs.
+- `source_columns` should reference the same `RecordDefinition` objects listed
+  in `records`.
+- `target_columns` should use MetaTable/SQLAlchemy column references such as
+  `Asset.uid`, not backend UID strings.
+- Do not ask users to provide FK names.
+- Do not ask users to provide `target_meta_table_uid`; the SDK resolves the
+  target MetaTable public `uid`.
+- FK hash material is source column names, target MetaTable public `uid`,
+  target column names, and `on_delete`.
+- FK hash material must not include generated names, backend database primary
+  keys, source-table FK row UIDs, backend projection/enforcement fields, target
+  storage hashes, or Python object/class repr values.
+- If FK target registration or MetaTable ownership is unclear, route to
+  `.agents/skills/mainsequence/data_publishing/meta_tables/SKILL.md`.
+
+### 9. Metadata is still required for production-quality nodes
+
+When the node is not a throwaway example, also provide table metadata through
+`DataNodeConfiguration.node_metadata` when a stable published identifier or
+description is needed.
+
+Use `json_schema_extra={"hash_excluded": True}` for descriptive metadata that
+must not rotate `update_hash` or `storage_hash`. Keep the older `runtime_only`
+marker only for legacy compatibility.
 
 ## Review Rules
 
@@ -153,7 +271,10 @@ When reviewing an existing DataNode, look for:
 
 - identifier collisions
 - accidental schema breaks
-- wrong meaning/scope/runtime-only split
+- wrong meaning/scope/hash-excluded split
+- missing `RecordDefinition` declarations
+- missing or incorrectly authored `SourceTableForeignKey` declarations when a
+  DataNode column references a MetaTable
 - misuse of `hash_namespace`
 - non-incremental `update()` behavior
 - hidden dependency creation inside `update()`
@@ -167,6 +288,10 @@ Do not claim success until you have checked:
 - the relevant docs were read first
 - the identifier choice is intentional
 - config fields are classified correctly
+- `DataNodeConfiguration.records` is present for new or materially edited nodes
+- declared `RecordDefinition` names and dtypes match the DataFrame returned by `update()`
+- any DataNode-to-MetaTable relationships use `SourceTableForeignKey` with
+  source record references and target column references
 - `dependencies()` is deterministic
 - `update()` is incremental
 - the DataFrame shape is valid

{mainsequence-4.0.3 → mainsequence-4.0.6}/agent_scaffold/skills/data_publishing/meta_tables/SKILL.md

@@ -15,7 +15,8 @@ This skill is for schema-driven application tables registered through TS Manager
 
 - define SQLAlchemy/Core or ORM table models for `MetaTable` registration
 - choose `platform_managed` or `external_registered` management mode
-- build registration requests from resolved SQLAlchemy metadata
+- register platform-managed tables through the model class API
+- build registration requests from resolved SQLAlchemy metadata when inspection is useful
 - define indexes and foreign keys in the table contract
 - design governed compiled SQL read and write operations
 - review table contracts for physical-name, namespace, and identifier issues
@@ -67,7 +68,7 @@ Before changing code, collect or infer:
 - expected read patterns
 - expected mutation patterns
 - whether TS Manager should create the physical table
-- the target `DynamicTableDataSource` UID
+- for `external_registered`, the target `DynamicTableDataSource` UID
 
 If ownership of the physical table lifecycle is unclear, stop before choosing a management mode.
 
@@ -92,15 +93,67 @@ Do not hand-build contract fragments when the SQLAlchemy helper can derive them.
 
 ### 2. Use storage-hash physical names for backend-managed tables
 
-For `platform_managed`, use `metatable_tablename(...)` as the SQLAlchemy `__tablename__`.
+For `platform_managed`, inherit from `PlatformManagedMetaTable`.
 
-The backend expects that deterministic physical name to match the registration storage hash.
+The mixin derives the SQLAlchemy physical table name from storage-relevant configuration and table shape. Do not hand-write `__tablename__` for normal backend-managed tables.
+
+Schema must come from SQLAlchemy table metadata, usually `__table_args__ = {"schema": "public"}` or the tuple form ending in `{"schema": ...}`. Do not add a separate MetaTable-specific schema attribute.
+
+Register through the class API:
+
+```python
+account_meta_table = Account.register(
+    description="Example account table.",
+    labels=["sdk-example"],
+)
+```
+
+For platform-managed registration, the data source is resolved from the active Main Sequence project/session, the same way DataNode does. Do not require or thread a `data_source_uid` through normal platform-managed example code.
+
+Only call `build_registration_request()` when the task explicitly needs to inspect or validate the payload before registration.
+
+For SDK examples, use a plain namespace constant:
+
+```python
+NAMESPACE = "sdk-examples"
+```
+
+Do not add an environment variable for namespace in examples.
+
+Do not add generic labels such as `"meta-table"` or `"platform-managed"` to examples. Keep labels specific to the example or domain.
+
+Do not add a `MAINSEQUENCE_META_TABLE_REGISTER` toggle in registration examples. Registration examples should register directly.
 
 ### 3. Register parent tables before child tables
 
 Foreign-key contracts reference the target `MetaTable` UID.
 
-Register parent tables first, then pass their UIDs when building child table registration requests.
+For `PlatformManagedMetaTable`, register parent tables first and then register child tables normally. The SDK inspects SQLAlchemy foreign-key constraints and resolves each target `MetaTable` by looking up the already registered table in the same data source, schema, and physical table name.
+
+Example registration order:
+
+```python
+account_meta_table = Account.register(...)
+asset_meta_table = Asset.register(...)
+```
+
+The child registration will fail if the parent table has not already been registered, because there is no target `MetaTable.uid` for the backend FK contract.
+
+Do not pass `target_meta_tables` or `target_meta_table_uid_by_fullname` in the normal platform-managed path. Use explicit FK target mappings only for edge cases where automatic lookup is ambiguous or impossible.
+
+For `external_registered`, there is no platform-managed parent lookup through the model class. Register the parent first, then build the child registration request with the parent UID mapped by target table fullname:
+
+```python
+account_meta_table = MetaTable.register(account_request)
+asset_request = external_registered_registration_request_from_sqlalchemy_model(
+    Asset,
+    data_source_uid=data_source_uid,
+    target_meta_table_uid_by_fullname={
+        Account.__table__.fullname: account_meta_table.uid,
+    },
+)
+asset_meta_table = MetaTable.register(asset_request)
+```
 
 ### 4. Governed operations declare scope
 
@@ -110,7 +163,7 @@ Do not execute unrestricted SQL outside the MetaTable operation contract.
 
 ### 5. Physical names come from registered resources
 
-Only use physical schema and table names returned by registered `MetaTable` objects when composing SQL strings.
+Only use physical table names returned by registered `MetaTable` objects when composing SQL strings.
 
 Do not hardcode platform-managed physical names manually.
 
@@ -119,9 +172,12 @@ Do not hardcode platform-managed physical names manually.
 When reviewing an existing MetaTable workflow, look for:
 
 - missing namespace or identifier
-- backend-managed models that do not use `metatable_tablename(...)`
+- backend-managed models that do not inherit `PlatformManagedMetaTable`
+- backend-managed examples that use namespace environment variables instead of a plain `sdk-examples` namespace
+- duplicate schema sources outside SQLAlchemy table metadata
 - external tables registered with unstable physical names
-- foreign keys that do not reference target MetaTable UIDs
+- platform-managed child tables that are registered before parent tables
+- external child registrations that do not map foreign-key targets to registered parent `MetaTable.uid` values
 - compiled SQL operations without complete table scope
 - raw SQL that hardcodes stale physical names
 - a table that should really be modeled as a DataNode instead
@@ -132,7 +188,7 @@ Do not claim success until you have checked:
 
 - the table contract matches the intended row contract
 - indexes are intentional
-- foreign keys point to the correct dependency targets
+- foreign keys resolve to the correct dependency targets
 - management mode is correct
 - backend-managed physical names match the storage hash
 - registration returns a `MetaTable.uid`
@@ -141,13 +197,14 @@ Do not claim success until you have checked:
 For related tables, also check:
 
 - aliases are readable
-- parent table UIDs are passed into child contracts
+- platform-managed parent tables are registered before child tables
+- external child registration requests map FK targets to the registered parent UIDs
 - query results still match the expected response contract
 
 ## This Skill Must Stop And Escalate When
 
 - physical table lifecycle ownership is unclear
-- the target data source is unknown
+- the target data source is unknown for an `external_registered` workflow
 - the task really requires a time-series published table
 - the workflow requires direct database credentials outside TS Manager governance
 - the task is actually an API or orchestration problem

{mainsequence-4.0.3 → mainsequence-4.0.6}/mainsequence/client/data_sources_interfaces/__init__.py

@@ -16,4 +16,10 @@ def get_duckdb_interface_class():
     return DuckDBInterface
 
 
-__all__ = ["get_duckdb_interface_class"]
+def get_sqlite_interface_class():
+    from .sqlite import SQLiteInterface
+
+    return SQLiteInterface
+
+
+__all__ = ["get_duckdb_interface_class", "get_sqlite_interface_class"]