mainsequence 4.0.16__tar.gz → 4.1.2__tar.gz

{mainsequence-4.0.16 → mainsequence-4.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.0.16
+Version: 4.1.2
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement
@@ -55,7 +55,6 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: cachetools
 Requires-Dist: click>=8.3.0
-Requires-Dist: cloudpickle
 Requires-Dist: concurrent-log-handler
 Requires-Dist: numpy
 Requires-Dist: opentelemetry-api
@@ -93,7 +92,7 @@ The Main Sequence Python SDK is the client and development toolkit for the Main
 
 The Main Sequence platform allows you to:
 
-1. rapidly build and deploy data products and data workflows as a unified API with a normalized structure through `DataNode`s
+1. rapidly build and deploy data products and data workflows as a unified API with normalized table contracts through `MetaTable`s and `DataNode`s
 2. rapidly deploy RBAC-enabled dashboards on the platform
 3. rapidly deploy agents using the Google Agent SDK
 
@@ -114,7 +113,7 @@ This repository contains the SDK and the documentation used to build and operate
 
 Main package areas:
 
-- `mainsequence.tdag`: data orchestration, `DataNode`s, update workflows, and persistence
+- `mainsequence.meta_tables`: `MetaTable`s, `DataNode`s, update workflows, and persistence
 - `mainsequence.client`: API client models for projects, jobs, data node storages, sharing, and platform resources
 - `mainsequence.cli`: the `mainsequence` command-line interface
 
@@ -137,10 +136,10 @@ Recommended entry points:
 
 - Tutorial:
   - [Setting a Project (CLI)](docs/tutorial/setting_a_project.md)
-  - [Creating a Data Node](docs/tutorial/creating_a_simple_data_node.md)
   - [Working With MetaTables](docs/tutorial/working_with_meta_tables.md)
-  - [Create Your First API](docs/tutorial/create_your_first_api.md)
+  - [Creating a Data Node](docs/tutorial/creating_a_simple_data_node.md)
   - [Role-Based Access Control](docs/tutorial/role_based_access_control.md)
+  - [Create Your First API](docs/tutorial/create_your_first_api.md)
   - [Turn Your Project Into an Agent](docs/tutorial/project_to_agent.md)
 - Knowledge:
   - [Data Nodes](docs/knowledge/data_nodes.md)
@@ -189,10 +188,10 @@ mainsequence project build_local_venv --path .
 
 From there, the normal learning path is:
 
-1. create your first `DataNode`
-2. model app-facing relational data with backend-managed `MetaTable`s when needed
-3. add an API or another application surface
-4. understand sharing and RBAC
+1. model your first canonical table with a backend-managed `MetaTable`
+2. create your first `DataNode` as an opinionated MetaTable-backed update workflow
+3. understand sharing and RBAC for published tables
+4. add an API or another application surface
 5. schedule jobs
 6. build dashboards or downstream consumers
 7. package the project as an agent-facing surface when the repository is ready

{mainsequence-4.0.16 → mainsequence-4.1.2}/README.md

@@ -13,7 +13,7 @@ The Main Sequence Python SDK is the client and development toolkit for the Main
 
 The Main Sequence platform allows you to:
 
-1. rapidly build and deploy data products and data workflows as a unified API with a normalized structure through `DataNode`s
+1. rapidly build and deploy data products and data workflows as a unified API with normalized table contracts through `MetaTable`s and `DataNode`s
 2. rapidly deploy RBAC-enabled dashboards on the platform
 3. rapidly deploy agents using the Google Agent SDK
 
@@ -34,7 +34,7 @@ This repository contains the SDK and the documentation used to build and operate
 
 Main package areas:
 
-- `mainsequence.tdag`: data orchestration, `DataNode`s, update workflows, and persistence
+- `mainsequence.meta_tables`: `MetaTable`s, `DataNode`s, update workflows, and persistence
 - `mainsequence.client`: API client models for projects, jobs, data node storages, sharing, and platform resources
 - `mainsequence.cli`: the `mainsequence` command-line interface
 
@@ -57,10 +57,10 @@ Recommended entry points:
 
 - Tutorial:
   - [Setting a Project (CLI)](docs/tutorial/setting_a_project.md)
-  - [Creating a Data Node](docs/tutorial/creating_a_simple_data_node.md)
   - [Working With MetaTables](docs/tutorial/working_with_meta_tables.md)
-  - [Create Your First API](docs/tutorial/create_your_first_api.md)
+  - [Creating a Data Node](docs/tutorial/creating_a_simple_data_node.md)
   - [Role-Based Access Control](docs/tutorial/role_based_access_control.md)
+  - [Create Your First API](docs/tutorial/create_your_first_api.md)
   - [Turn Your Project Into an Agent](docs/tutorial/project_to_agent.md)
 - Knowledge:
   - [Data Nodes](docs/knowledge/data_nodes.md)
@@ -109,10 +109,10 @@ mainsequence project build_local_venv --path .
 
 From there, the normal learning path is:
 
-1. create your first `DataNode`
-2. model app-facing relational data with backend-managed `MetaTable`s when needed
-3. add an API or another application surface
-4. understand sharing and RBAC
+1. model your first canonical table with a backend-managed `MetaTable`
+2. create your first `DataNode` as an opinionated MetaTable-backed update workflow
+3. understand sharing and RBAC for published tables
+4. add an API or another application surface
 5. schedule jobs
 6. build dashboards or downstream consumers
 7. package the project as an agent-facing surface when the repository is ready

mainsequence-4.1.2/agent_scaffold/skills/data_publishing/data_nodes/SKILL.md

@@ -0,0 +1,380 @@
+---
+name: mainsequence-data-nodes
+description: Use this skill when the task is about producing, changing, validating, or reviewing Main Sequence DataNode update processes. This skill owns DataNode update configuration, dependencies, update logic, hashing, namespaces, and validation against a PlatformTimeIndexMetaData storage contract. It does not own generic MetaTable governance, API route contracts, scheduling, sharing policy, or storage registration internals.
+---
+
+# Main Sequence Data Nodes
+
+## Overview
+
+Use this skill when the task changes a DataNode producer.
+
+A DataNode is an update process. It is not the canonical storage model. Storage
+is defined by a registered `PlatformTimeIndexMetaData` SQLAlchemy model.
+
+Canonical workflow:
+
+1. Define a `PlatformTimeIndexMetaData` storage class.
+2. Register or bind that storage class before constructing the DataNode.
+3. Construct the DataNode with `config=...` and `storage_table=StorageClass`.
+4. Return a DataFrame from `update()` that matches the storage class contract.
+
+## This Skill Can Do
+
+- create a new `DataNode` update process
+- modify an existing `DataNode` update process
+- review whether a DataNode change affects update identity or table contract
+- define or refactor `DataNodeConfiguration`
+- classify config fields into update identity and hash-excluded descriptive metadata
+- implement or review:
+  - `dependencies()`
+  - `update()`
+  - `prepare_update_statistics()`
+  - `get_asset_list()` when the update is asset scoped
+- design single-index or `(time_index, unique_identifier)` DataFrame outputs
+- validate output shape against a `PlatformTimeIndexMetaData` storage contract
+- define explicit `hash_namespace(...)` validation strategy
+- write or review DataNode smoke tests
+- decide whether a consumer should use `APIDataNode`
+
+## This Skill Must Not Claim
+
+This skill must not claim ownership of:
+
+- generic MetaTable registration or governed operation semantics
+- storage creation inside `DataNode` or `PersistManager`
+- HTTP route design or FastAPI response contracts
+- workspace/widget layout payloads
+- job creation, scheduling, image pinning, or release creation
+- RBAC or sharing policy
+- domain strategy semantics
+
+If the task depends on one of those areas, route it explicitly instead of guessing.
+
+## Route Adjacent Work
+
+- MetaTables and storage contracts:
+  `.agents/skills/mainsequence/data_publishing/meta_tables/SKILL.md`
+- APIs and FastAPI:
+  `.agents/skills/mainsequence/application_surfaces/api_surfaces/SKILL.md`
+- Command Center workspaces:
+  `.agents/skills/mainsequence/command_center/workspace_builder/SKILL.md`
+- AppComponents and custom forms:
+  `.agents/skills/mainsequence/command_center/app_components/SKILL.md`
+- Jobs, images, resources, and releases:
+  `.agents/skills/mainsequence/platform_operations/orchestration_and_releases/SKILL.md`
+- RBAC and sharing:
+  `.agents/skills/mainsequence/platform_operations/access_control_and_sharing/SKILL.md`
+
+## Read First
+
+1. `docs/knowledge/data_nodes.md`
+2. `docs/knowledge/meta_tables/sqlalchemy.md`
+
+## Inputs This Skill Needs
+
+Before changing code, collect or infer:
+
+- dataset meaning
+- registered `PlatformTimeIndexMetaData` storage class or the class to create
+- expected time index and identity index shape
+- expected columns and dtypes from the storage class
+- upstream dependencies
+- whether the update is asset scoped
+- first-run or backfill bounds
+- whether the change must preserve the existing table contract
+
+If one of these is unknown and changes the contract, stop and resolve it before
+implementation.
+
+## Required Decisions
+
+For every non-trivial DataNode task, make these decisions explicitly:
+
+1. Is this a new dataset or the same dataset?
+2. Is this change storage-contract work or update-process work?
+3. Is the storage class already registered or should the MetaTable skill handle it?
+4. Is the node single-index or MultiIndex?
+5. Does the first validation run happen under an explicit `hash_namespace(...)`?
+
+## Build Rules
+
+### 1. Treat Storage As A PlatformTimeIndexMetaData Contract
+
+The following are storage-contract decisions:
+
+- table namespace and identifier
+- SQLAlchemy columns and dtypes
+- time index name
+- identity index names
+- foreign keys
+- table description and labels
+
+Do not put those concerns in `DataNodeConfiguration`.
+
+Minimal pattern:
+
+```python
+import datetime
+
+from sqlalchemy import DateTime, Float, MetaData
+from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
+
+from mainsequence.meta_tables import PlatformTimeIndexMetaData
+
+
+class Base(DeclarativeBase):
+    metadata = MetaData()
+
+
+class PricesTable(PlatformTimeIndexMetaData, Base):
+    __metatable_namespace__ = "<domain_namespace>"
+    __metatable_identifier__ = "<table_identifier>"
+    __time_index_name__ = "time_index"
+    __index_names__ = ["time_index", "unique_identifier"]
+
+    time_index: Mapped[datetime.datetime] = mapped_column(
+        DateTime(timezone=True),
+        nullable=False,
+    )
+    unique_identifier: Mapped[str] = mapped_column(nullable=False)
+    close: Mapped[float] = mapped_column(Float, nullable=False)
+```
+
+Register or bind storage before constructing the DataNode:
+
+```python
+PricesTable.register(data_source_uid=data_source_uid)
+```
+
+or:
+
+```python
+PricesTable.bind_meta_table(existing_meta_table)
+```
+
+### 2. Keep DataNode As Update Logic
+
+The DataNode constructor should accept:
+
+- a `DataNodeConfiguration`
+- a registered `storage_table: type[PlatformTimeIndexMetaData]`
+- optional `hash_namespace`
+
+The constructor `storage_table` is the output storage contract. Keep it out of
+`DataNodeConfiguration`.
+
+If the DataNode needs to select another DataNode's storage table as a
+dependency, put that dependency storage reference in the config as
+`type[PlatformTimeIndexMetaData]`. Do not add an extra constructor argument for
+dependency storage tables. Config values of this type are hashed by the bound
+`TimeIndexMetaData.uid` from `StorageClass.__time_index_metadata__`, so the class
+must be registered or bound before DataNode construction.
+
+Do not accept `test_node`. It has been removed. Use explicit
+`hash_namespace(...)` or `hash_namespace="..."`.
+
+Pattern:
+
+```python
+from mainsequence.meta_tables import DataNode, DataNodeConfiguration
+from mainsequence.meta_tables import PlatformTimeIndexMetaData
+
+
+class PricesConfig(DataNodeConfiguration):
+    shard_id: str
+
+
+class PricesUpdate(DataNode):
+    def __init__(
+        self,
+        config: PricesConfig,
+        storage_table: type[PlatformTimeIndexMetaData],
+        *,
+        hash_namespace: str | None = None,
+    ):
+        self.config = config
+        super().__init__(
+            config=config,
+            storage_table=storage_table,
+            hash_namespace=hash_namespace,
+        )
+
+    def dependencies(self):
+        return {}
+
+    def update(self):
+        ...
+```
+
+### 3. Configuration Is Update-Scoped By Default
+
+Every `DataNodeConfiguration` field participates in `update_hash` by default.
+
+Do not use:
+
+- `json_schema_extra={"update_only": True}`
+- `json_schema_extra={"runtime_only": True}`
+- `json_schema_extra={"ignore_from_storage_hash": True}`
+- `_ARGS_IGNORE_IN_STORAGE_HASH`
+
+Those are removed. The only supported opt-out is:
+
+```python
+Field(..., json_schema_extra={"hash_excluded": True})
+```
+
+Use `hash_excluded` only for descriptive metadata that must not affect update
+identity. If a field changes output values, dependencies, source choice, or
+updater scope, it must remain a normal config field.
+
+### 4. `hash_namespace` Is Isolation Only
+
+Use explicit `hash_namespace(...)` for:
+
+- namespaced tests
+- isolated experimentation
+- shared-backend safety
+
+Do not use namespace to encode business meaning.
+
+Do not use `test_node=True`; it is not supported.
+
+### 5. `update()` Should Be Incremental By Default
+
+Use `UpdateStatistics`.
+
+Do not fetch or return full history every run unless there is a documented
+reason.
+
+### 6. `time_index` Must Be Nanosecond UTC
+
+Every non-empty DataFrame returned by `update()` must have its first index
+level named `time_index` with dtype exactly `datetime64[ns, UTC]`.
+
+Do not rely on `pd.Timestamp.now("UTC").normalize()` or
+`pd.Timestamp.now("UTC").floor(...)` alone. Some pandas versions preserve
+microsecond resolution and produce `datetime64[us, UTC]`, which the DataNode
+runtime rejects.
+
+For a single-index DataFrame, construct the index with an explicit dtype:
+
+```python
+time_index = pd.Timestamp.now("UTC").normalize()
+return pd.DataFrame(
+    {"close": [100.0]},
+    index=pd.DatetimeIndex(
+        [time_index],
+        name="time_index",
+        dtype="datetime64[ns, UTC]",
+    ),
+)
+```
+
+For a MultiIndex DataFrame, construct the time level separately:
+
+```python
+time_index = pd.DatetimeIndex(
+    [row[0] for row in rows],
+    dtype="datetime64[ns, UTC]",
+)
+index = pd.MultiIndex.from_arrays(
+    [
+        time_index,
+        [row[1] for row in rows],
+    ],
+    names=["time_index", "unique_identifier"],
+)
+```
+
+The validator error to prevent is:
+
+```text
+Time index must be datetime64[ns, UTC]
+```
+
+### 7. Dependencies Must Be Deterministic
+
+Dependencies belong in constructor setup and `dependencies()`.
+
+Dependency storage-table selection belongs in `DataNodeConfiguration`, because
+changing it changes the dependency graph and update identity.
+
+Do not construct dependency graphs dynamically inside `update()`.
+
+### 8. Asset-Scoped Updates Must Be Explicit
+
+If the node emits `(time_index, unique_identifier)`:
+
+- `unique_identifier` should represent an Asset identity
+- asset scope should live in update configuration when it affects the updater
+- `get_asset_list()` must reflect the effective updater asset scope when the
+  workflow uses that hook
+- missing assets should be resolved or registered by the relevant workflow
+
+### 9. Foreign Keys Belong To The Storage Contract
+
+For new code, model foreign keys on the `PlatformTimeIndexMetaData` storage
+class or route the storage-contract work to the MetaTable skill.
+
+Do not add DataNode configuration fields just to mutate storage metadata.
+
+### 10. Metadata Belongs To Storage
+
+Production-quality table identifiers, descriptions, labels, column docs, and
+foreign-key metadata belong to the storage class/MetaTable registration path.
+
+Do not use `DataNodeMetaData` or `RecordDefinition` as the canonical schema
+surface for new DataNode work.
+
+## Review Rules
+
+When reviewing an existing DataNode, look for:
+
+- output storage contract hidden in `DataNodeConfiguration`
+- dependency storage table passed as an ad hoc constructor argument
+- old `RecordDefinition` or `DataNodeMetaData` schema patterns
+- `update_only`, `runtime_only`, or `ignore_from_storage_hash`
+- `test_node=True`
+- missing explicit `storage_table`
+- accidental storage registration inside the DataNode
+- wrong meaning/scope/hash-excluded split
+- misuse of `hash_namespace`
+- non-incremental `update()` behavior
+- hidden dependency creation inside `update()`
+- invalid asset-indexed output shape
+- `time_index` dtype that is not exactly `datetime64[ns, UTC]`
+- DataFrame columns that do not match the `PlatformTimeIndexMetaData` class
+
+## Validation Checklist
+
+Do not claim success until you have checked:
+
+- the relevant docs were read first
+- storage is a registered or bound `PlatformTimeIndexMetaData` class
+- the DataNode constructor requires `storage_table`
+- dependency storage-table references live in config and are registered or bound
+- config fields are updater-scoped by default
+- no removed hash metadata markers remain
+- no `test_node` usage remains
+- `dependencies()` is deterministic
+- `update()` is incremental
+- the DataFrame shape matches the storage class
+- non-empty outputs have first index level `time_index` with dtype `datetime64[ns, UTC]`
+- the first validation run uses explicit `hash_namespace(...)` when it touches a shared backend
+
+For asset-scoped updates, also check:
+
+- `get_asset_list()` is correct when used
+- no duplicate `(time_index, unique_identifier)` rows are emitted
+- assets exist or are registered idempotently when needed
+
+## This Skill Must Stop And Escalate When
+
+- the change may break an existing published table contract and the versioning decision is unclear
+- the intended storage class or MetaTable registration path is unclear
+- the node needs asset identities but the asset-resolution strategy is unclear
+- the task is actually an API, MetaTable, orchestration, or sharing problem
+- docs, skill instructions, and code disagree on hashing or runtime behavior
+
+Do not guess through contract changes.