mainsequence 4.1.16__tar.gz → 4.1.18__tar.gz

{mainsequence-4.1.16/mainsequence.egg-info → mainsequence-4.1.18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.1.16
+Version: 4.1.18
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement
@@ -53,6 +53,7 @@ Classifier: Operating System :: OS Independent
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: alembic>=1.18.4
 Requires-Dist: cachetools
 Requires-Dist: click>=8.3.0
 Requires-Dist: concurrent-log-handler

{mainsequence-4.1.16 → mainsequence-4.1.18}/agent_scaffold/skills/data_publishing/data_nodes/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: mainsequence-data-nodes
-description: Use this skill when the task is about producing, changing, validating, or reviewing Main Sequence DataNode update processes. This skill owns DataNode update configuration, dependencies, update logic, hashing, namespaces, and validation against a PlatformTimeIndexMetaData or MigrationManagedTimeIndexMetaData storage contract. It does not own generic MetaTable governance, API route contracts, scheduling, sharing policy, or storage registration internals.
+description: Use this skill when the task is about producing, changing, validating, or reviewing Main Sequence DataNode update processes. This skill owns DataNode update configuration, dependencies, update logic, hashing, namespaces, and validation against a PlatformTimeIndexMetaData storage contract. It does not own generic MetaTable governance, Alembic migration execution, API route contracts, scheduling, sharing policy, or storage registration internals.
 ---
 
 # Main Sequence Data Nodes
@@ -10,14 +10,12 @@ description: Use this skill when the task is about producing, changing, validati
 Use this skill when the task changes a DataNode producer.
 
 A DataNode is an update process. It is not the canonical storage model. Storage
-is defined by a `PlatformTimeIndexMetaData` SQLAlchemy model, or by
-`MigrationManagedTimeIndexMetaData` when the storage table must support
-in-place contract migrations.
+is defined by a `PlatformTimeIndexMetaData` SQLAlchemy model. Physical schema
+evolution is handled with Alembic, not a DataNode storage subclass.
 
 Canonical workflow:
 
-1. Define a `PlatformTimeIndexMetaData` storage class, or
-   `MigrationManagedTimeIndexMetaData` for migration-managed storage.
+1. Define a `PlatformTimeIndexMetaData` storage class.
 2. Construct the DataNode with `config=...` and `storage_table=StorageClass`.
 3. Let the SDK register the output storage class automatically when needed.
 4. Return a DataFrame from `update()` that matches the storage class contract.
@@ -34,8 +32,7 @@ Canonical workflow:
   - `update()`
   - `prepare_update_statistics()`
 - design single-index or multidimensional time-first DataFrame outputs
-- validate output shape against a `PlatformTimeIndexMetaData` or
-  `MigrationManagedTimeIndexMetaData` storage contract
+- validate output shape against a `PlatformTimeIndexMetaData` storage contract
 - define explicit `hash_namespace(...)` validation strategy
 - write or review DataNode smoke tests
 - decide whether a consumer should use `APIDataNode`
@@ -80,8 +77,7 @@ If the task depends on one of those areas, route it explicitly instead of guessi
 Before changing code, collect or infer:
 
 - dataset meaning
-- `PlatformTimeIndexMetaData` or `MigrationManagedTimeIndexMetaData` output
-  storage class, or the class to create
+- `PlatformTimeIndexMetaData` output storage class, or the class to create
 - expected time index and identity index shape
 - expected columns and dtypes from the storage class
 - upstream dependencies
@@ -195,8 +191,7 @@ during bootstrap when code needs the returned metadata object:
 PricesTable.register()
 ```
 
-`PlatformTimeIndexMetaData.register()` and
-`MigrationManagedTimeIndexMetaData.register()` are the storage lifecycle paths.
+`PlatformTimeIndexMetaData.register()` is the storage lifecycle path.
 Treat them as idempotent get-or-create operations: the platform returns the
 registered metadata and UID, and the SDK records that metadata on the class. Do
 not manually attach an existing UID, reconstruct a generic `MetaTable`, or use
@@ -215,16 +210,13 @@ The constructor `storage_table` is the output storage contract. Keep it out of
 construct the node; `DataNode` and `PersistManager` call the SDK registration
 lifecycle automatically when the class is not yet bound.
 
-`MigrationManagedTimeIndexMetaData` subclasses `PlatformTimeIndexMetaData`, so
-it is valid anywhere the DataNode API expects `type[PlatformTimeIndexMetaData]`.
-
 If the DataNode needs to select another DataNode's storage table as a
 dependency, put that dependency storage reference in the config as
 `type[PlatformTimeIndexMetaData]`. Do not add an extra constructor argument for
 dependency storage tables. Config values of this type are hashed by the bound
-`TimeIndexMetaData.uid` from `StorageClass.__time_index_metadata__`; this also
-applies to `MigrationManagedTimeIndexMetaData`. If the class is not yet bound,
-the config serializer calls `StorageClass.register()` before reading the UID.
+`TimeIndexMetaData.uid` from `StorageClass.__time_index_metadata__`. If the
+class is not yet bound, the config serializer calls `StorageClass.register()`
+before reading the UID.
 
 Do not accept `test_node`. It has been removed. Use explicit
 `hash_namespace(...)` or `hash_namespace="..."`.
@@ -379,15 +371,10 @@ The validator error to prevent is:
 Time index must be datetime64[ns, UTC]
 ```
 
-Use `MigrationManagedTimeIndexMetaData` from the first version when a DataNode
-storage table must support in-place contract migrations through a
-`MigrationMetaTable` registry. It subclasses `PlatformTimeIndexMetaData`, keeps
-the TimeIndexMetaData registration endpoint and time-index validation, and uses
-stable identifier-addressed storage identity so contract hashes can rotate.
-
-Do not switch an already published shape-addressed `PlatformTimeIndexMetaData`
-table to migration-managed identity without an adoption plan for the old
-MetaTable UID/storage hash.
+Use Alembic from the first version when a DataNode storage table must support
+physical schema evolution. Keep the `PlatformTimeIndexMetaData` catalog model as
+the SDK storage contract, apply Alembic-rendered SQL through the migration
+workflow, then register or refresh the MetaTable catalog binding separately.
 
 ### 7. Dependencies Must Be Deterministic
 
@@ -400,10 +387,9 @@ Do not construct dependency graphs dynamically inside `update()`.
 
 ### 8. Foreign Keys Belong To The Storage Contract
 
-For new code, model foreign keys on the `PlatformTimeIndexMetaData` or
-`MigrationManagedTimeIndexMetaData` storage class, or route the
-storage-contract work to the MetaTable skill. When a DataNode storage table
-needs a platform-managed FK, use
+For new code, model foreign keys on the `PlatformTimeIndexMetaData` storage
+class, or route the storage-contract work to the MetaTable skill. When a
+DataNode storage table needs a platform-managed FK, use
 `MetaTableForeignKey(TargetModel, column=...)` on the storage class. Do not use
 `ForeignKey(Target.__table__.c.uid)`, table fullnames, or explicit target UID
 maps in DataNode examples.
@@ -445,16 +431,14 @@ When reviewing an existing DataNode, look for:
 - hidden dependency creation inside `update()`
 - invalid identity-indexed output shape
 - `time_index` dtype that is not exactly `datetime64[ns, UTC]`
-- DataFrame columns that do not match the `PlatformTimeIndexMetaData` or
-  `MigrationManagedTimeIndexMetaData` class
+- DataFrame columns that do not match the `PlatformTimeIndexMetaData` class
 
 ## Validation Checklist
 
 Do not claim success until you have checked:
 
 - the relevant docs were read first
-- output storage is a `PlatformTimeIndexMetaData` or
-  `MigrationManagedTimeIndexMetaData` class that the SDK can register
+- output storage is a `PlatformTimeIndexMetaData` class that the SDK can register
 - storage has an intention-rich `__metatable_description__`
 - every storage column has an intention-rich `info.description`
 - the DataNode constructor requires `storage_table`

{mainsequence-4.1.16 → mainsequence-4.1.18}/agent_scaffold/skills/data_publishing/meta_tables/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: mainsequence-meta-tables
-description: Use this skill when the task is about defining, registering, querying, migrating, or reviewing Main Sequence MetaTables. This skill owns SQLAlchemy table contracts, backend-managed table registration, external table registration, client-defined MetaTable migration registries, governed compiled SQL operations, foreign keys, indexes, and validation rules. It does not own DataNode producers, API route contracts, scheduling, releases, or sharing policy.
+description: Use this skill when the task is about defining, registering, querying, migrating, or reviewing Main Sequence MetaTables. This skill owns SQLAlchemy table contracts, backend-managed table registration, external table registration, Alembic-based MetaTable migrations, governed compiled SQL operations, foreign keys, indexes, and validation rules. It does not own DataNode producers, API route contracts, scheduling, releases, or sharing policy.
 ---
 
 # Main Sequence MetaTables
@@ -19,8 +19,8 @@ This skill is for schema-driven application tables registered through TS Manager
 - build registration requests from resolved SQLAlchemy metadata when inspection is useful
 - define indexes and foreign keys in the table contract
 - design governed compiled SQL read and write operations
-- design client-defined `MigrationMetaTable` registries for contract evolution
-- package SQL migrations and old/new contract hashes for backend apply
+- design provider-based Alembic contract evolution for MetaTables
+- run the documented `mainsequence migrations ...` lifecycle for Alembic-backed MetaTable changes
 - review table contracts for physical-name, namespace, and identifier issues
 - review whether a task should be a `MetaTable` or a `DataNode`
 
@@ -55,11 +55,12 @@ If the user is still in the discovery process and does not yet know what data ex
 ## Read First
 
 1. `docs/tutorial/working_with_meta_tables.md`
-2. `docs/knowledge/meta_tables/index.md`
-3. `docs/knowledge/meta_tables/sqlalchemy.md`
-4. `docs/knowledge/meta_tables/compiled_sql.md`
-5. `docs/knowledge/meta_tables/migrations.md`
-6. `docs/knowledge/meta_tables/api.md`
+2. For migration work, `docs/tutorial/metatable_migrations.md`
+3. `docs/knowledge/meta_tables/index.md`
+4. `docs/knowledge/meta_tables/sqlalchemy.md`
+5. `docs/knowledge/meta_tables/compiled_sql.md`
+6. `docs/knowledge/meta_tables/migrations.md`
+7. `docs/knowledge/meta_tables/api.md`
 
 ## Inputs This Skill Needs
 
@@ -72,8 +73,9 @@ Before changing code, collect or infer:
 - expected mutation patterns
 - whether TS Manager should create the physical table
 - for `external_registered`, the target `DynamicTableDataSource` UID
-- for contract changes, the target migration registry MetaTable or registry class
-- for contract changes, the package, migration namespace, revision, parent revision, SQL file, affected table identifiers, and old/new SQLAlchemy contract declarations
+- for contract changes, the selected `AlembicMetaTableMigration` provider or provider module path
+- for contract changes, the provider's `AlembicVersionMetaTable` binding and whether it has been registered
+- for contract changes, the intended Alembic revision, parent/current revision, target revision, and updated SQLAlchemy declarations
 
 If ownership of the physical table lifecycle is unclear, stop before choosing a management mode.
 
@@ -87,8 +89,9 @@ For every non-trivial task, decide:
 4. What namespace and identifier define the logical table identity?
 5. Are foreign-key dependencies aligned with registration order?
 6. What governed operations should be allowed by the declared table scope?
-7. If the table already exists and its contract changes, is this a migration through a registry row rather than normal registration?
-8. For a migration, what stable affected-table identifiers and old/new contract hashes will the backend validate?
+7. If the table already exists and its contract changes, is this an Alembic migration rather than normal registration?
+8. For a migration, which provider object controls `target_metadata`, `script_location`, `alembic_registry`, and `metatable_models`?
+9. For a migration, has the provider's Alembic version-table binding been registered?
 
 ## Build Rules
 
@@ -105,9 +108,9 @@ For `platform_managed`, inherit from `PlatformManagedMetaTable`.
 The mixin derives the SQLAlchemy physical table name from storage-relevant configuration and table shape. Do not hand-write `__tablename__` for normal backend-managed tables.
 
 When a platform-managed table must support in-place contract migrations from its
-first version, use `MigrationManagedMetaTable` instead. For time-indexed
-DataNode storage that must support in-place contract migrations, use
-`MigrationManagedTimeIndexMetaData`.
+first version, use Alembic. Keep the SDK model as a normal
+`PlatformManagedMetaTable` or `PlatformTimeIndexMetaData` catalog contract, and
+apply physical schema changes through the Alembic migration workflow.
 
 Schema must come from SQLAlchemy table metadata, usually `__table_args__ = {"schema": "public"}` or the tuple form ending in `{"schema": ...}`. Do not add a separate MetaTable-specific schema attribute.
 
@@ -237,7 +240,11 @@ asset_request = external_registered_registration_request_from_sqlalchemy_model(
 asset_meta_table = MetaTable.register(asset_request)
 ```
 
-### 4. Schema changes use a migration registry
+### 4. Schema changes use Alembic
+
+When doing migration work, first read
+`docs/tutorial/metatable_migrations.md`. That document is the tutorial source
+for the provider-based Alembic lifecycle.
 
 Do not apply in-place contract changes by changing a `PlatformManagedMetaTable`
 SQLAlchemy class and calling normal registration again. Shape-addressed
@@ -245,73 +252,38 @@ SQLAlchemy class and calling normal registration again. Shape-addressed
 foreign keys, or constraints change, so new code cannot reliably recover the
 previous shape-derived table.
 
-For contract evolution, use `mainsequence.meta_tables.migrations`:
-
-- declare a client-owned `MigrationMetaTable` registry
-- load SQL and manifest files from the installed Python package
-- compute `manifest_sha256` and `sql_sha256`
-- compute `old_contract_hashes`, `new_contract_hashes`, and `new_contracts`
-  from SQLAlchemy model declarations
-- sync the migration row into the registry MetaTable
-- call `MetaTable.apply_migration(...)` with a `metatable-migration.v1`
-  request that references the registry row
-
-`migration_meta_table_uid` is the UID of the registry MetaTable that stores
-migration rows. It is not the UID of the table being migrated.
-
-Affected tables are resolved by stable logical identifiers:
-
-```python
-affected_tables = [{"identifier": "sdk-examples.Asset"}]
-```
-
-For a changed table, model the old and new contracts explicitly and keep the
-same `__metatable_identifier__` on both declarations:
-
-```python
-from mainsequence.meta_tables import MigrationManagedMetaTable
-
-
-class AssetBeforeMigration(MigrationManagedMetaTable, BeforeBase):
-    __metatable_namespace__ = "sdk-examples"
-    __metatable_identifier__ = "sdk-examples.Asset"
-
-    uid: Mapped[uuid.UUID] = mapped_column(Uuid, primary_key=True)
-    symbol: Mapped[str] = mapped_column(String(64), nullable=False)
-
-
-class AssetAfterMigration(MigrationManagedMetaTable, AfterBase):
-    __metatable_namespace__ = "sdk-examples"
-    __metatable_identifier__ = "sdk-examples.Asset"
-
-    uid: Mapped[uuid.UUID] = mapped_column(Uuid, primary_key=True)
-    symbol: Mapped[str] = mapped_column(String(64), nullable=False)
-    status: Mapped[str] = mapped_column(String(32), nullable=False)
+For contract evolution, define or update one selected
+`AlembicMetaTableMigration` provider:
+
+- put the provider in `mainsequence_migrations.py:migration` or pass
+  `--provider module.path:migration`
+- set `package`, `migration_namespace`, `script_location`, and `target_metadata`
+- set `alembic_registry` to an `AlembicVersionMetaTable` subclass
+- list the post-apply catalog scope in `metatable_models`
+- generate, render, dry-run, apply, and optionally refresh catalog bindings
+  through `mainsequence migrations ...` commands
+
+`alembic_version_meta_table_uid` is the UID of the catalog binding for Alembic's
+version table. It is not the UID of the table being migrated.
+
+Do not ask users to construct backend migration payloads, call low-level
+migration request models, or use SDK helper functions directly. The backend
+request shape is reference material in the tutorial; the user-facing path is:
+
+```bash
+mainsequence migrations register-version-table --provider mainsequence_migrations:migration
+mainsequence migrations revision --provider mainsequence_migrations:migration --autogenerate -m "change"
+mainsequence migrations render --provider mainsequence_migrations:migration --to head
+mainsequence migrations upgrade --provider mainsequence_migrations:migration --to head --dry-run
+mainsequence migrations upgrade --provider mainsequence_migrations:migration --to head --apply --register-metatables
 ```
 
-Then package the migration row with both contract versions:
-
-```python
-packaged = load_packaged_migration(
-    "examples.meta_tables.migrations",
-    "packaged/002_add_asset_status.yaml",
-    old_contract_models={"sdk-examples.Asset": AssetBeforeMigration},
-    new_contract_models={"sdk-examples.Asset": AssetAfterMigration},
-)
-```
-
-The backend apply endpoint validates the old hash before SQL execution and the
-new hash after introspection/MetaTable refresh. The SDK must not send executable
-SQL directly in the apply request body; the backend reads SQL from the
-referenced registry row.
-
-For time-indexed DataNode storage, use `MigrationManagedTimeIndexMetaData`. It
-is a `MigrationManagedMetaTable` target for packaging and validation, but it
-keeps the `TimeIndexMetaData` registration endpoint, `time_index_name`,
-`index_names`, and time-indexed table contract.
+The SQL must be Alembic-rendered from the selected provider. After SQL apply
+succeeds, register or refresh only the application MetaTable catalog bindings
+listed in `migration.metatable_models`.
 
-Use `examples/meta_tables/migrations/contract_hash_rotation.py` as the canonical
-example for a MetaTable contract change.
+Do not use SDK-managed migration artifact tables, artifact sync helpers, or custom
+`operations()` migration modules.
 
 ### 5. Governed operations declare scope
 
@@ -333,18 +305,16 @@ When reviewing an existing MetaTable workflow, look for:
 - missing `__metatable_description__`, or a description that only repeats column names instead of table intention
 - mapped columns without `info.label` and `info.description`
 - backend-managed models that do not inherit `PlatformManagedMetaTable`
-- migration-managed tables that should use `MigrationManagedMetaTable` or
-  `MigrationManagedTimeIndexMetaData` from their first version
+- schema changes that bypass Alembic or try to use SDK operation lists
+- migration work that lacks a selected `AlembicMetaTableMigration` provider
 - backend-managed examples that use namespace environment variables instead of a plain `sdk-examples` namespace
 - duplicate schema sources outside SQLAlchemy table metadata
 - external tables registered with unstable physical names
 - platform-managed examples that manually sequence parent registration instead
   of relying on `MetaTableForeignKey(...)` recursive registration
 - external child registrations that do not map foreign-key targets to registered parent `MetaTable.uid` values
-- contract changes attempted through normal registration instead of a `MigrationMetaTable` registry row
-- migration apply requests that confuse `migration_meta_table_uid` with an affected table UID
-- migration rows missing old/new contract hashes for affected in-place tables
-- migration rows that include arbitrary SQL in the request body instead of packaged SQL in the registry row
+- contract changes attempted through normal registration instead of an Alembic migration
+- migration work that asks users to define backend payloads, artifact rows, or SDK request objects
 - compiled SQL operations without complete table scope
 - raw SQL that hardcodes stale physical names
 - a table that should really be modeled as a DataNode instead
@@ -362,10 +332,11 @@ Do not claim success until you have checked:
 - backend-managed physical names match the storage hash
 - registration returns a `MetaTable.uid`
 - compiled SQL operations declare table scope
-- migrations use a client-defined `MigrationMetaTable` registry
-- migration rows include stable affected table identifiers
-- migration rows include old/new contract hashes and post-migration contracts
-- migration apply/status helpers use typed `metatable-migration.v1` request and response models
+- migrations use Alembic-rendered SQL
+- migrations are scoped by an `AlembicMetaTableMigration` provider
+- the provider's Alembic version-table binding is registered before apply/current
+- post-apply catalog registration is scoped to `migration.metatable_models`
+- user-facing migration instructions stay on the documented CLI/provider lifecycle
 
 For related tables, also check:
 
@@ -381,8 +352,8 @@ For related tables, also check:
 - the target data source is unknown for an `external_registered` workflow
 - the task really requires a time-series published table
 - the workflow requires direct database credentials outside TS Manager governance
-- a requested contract change lacks a stable affected table identifier or old contract anchor
-- the user expects the SDK to autogenerate migrations or parse arbitrary request-body SQL
+- a requested contract change lacks a selected provider or registered Alembic version-table binding
+- the user expects the SDK to invent schema changes without Alembic/provider metadata
 - the task is actually an API or orchestration problem
 
 Do not guess through registration or execution semantics.