mainsequence 4.3.8__tar.gz → 4.3.14__tar.gz

{mainsequence-4.3.8/mainsequence.egg-info → mainsequence-4.3.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.3.8
+Version: 4.3.14
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement

mainsequence-4.3.14/agent_scaffold/skills/data_publishing/meta_table_migrations/SKILL.md

@@ -0,0 +1,102 @@
+---
+name: mainsequence-metatable-migrations
+description: Use this skill when the task is about Main Sequence MetaTable schema evolution, Alembic provider setup, migration scaffold generation, revision creation, migration upgrade/current commands, provider-scoped MetaTable reservation/finalization, or debugging migration lifecycle failures. This skill owns the migration-first workflow for platform-managed MetaTables and should be used instead of direct MetaTable.register() guidance.
+---
+
+# Main Sequence MetaTable Migrations
+
+## Overview
+
+Use this skill for platform-managed MetaTable schema lifecycle work. The
+standard path is migration-first: declare SQLAlchemy MetaTable models, put them
+in a provider, run `mainsequence migrations ...`, and let the SDK reserve,
+migrate, finalize, and bind provider-scoped MetaTables.
+
+Do not recommend direct `MetaTable.register()` or model `.register()` calls for
+platform-managed application tables outside the migration workflow.
+
+## Read First
+
+1. `docs/tutorial/metatable_migrations.md`
+2. `docs/knowledge/meta_tables/migrations.md`
+3. `docs/knowledge/meta_tables/api.md`
+4. `docs/adr/0020-metatable-migration-artifact-registry.md`
+5. `docs/adr/0026-sdk-owned-migration-scaffolding.md`
+
+## Required Decisions
+
+- Which provider module owns the migration stream?
+- Which package and migration namespace identify the provider?
+- Which `AlembicVersionMetaTable` binding points to Alembic's version table?
+- Which SQLAlchemy `MetaData` object is the target metadata?
+- Which provider-scoped MetaTable models belong in `metatable_models`?
+- Does a dynamic provider need `metadata_for_models(...)` instead of full
+  package metadata?
+- Does the project need an `after_register_metatables` catalog hook, and does
+  that hook use `context.metatable_models` and `context.registered_metatables`
+  instead of importing a broader registry?
+
+## Standard Workflow
+
+Use SDK-owned scaffold and helpers when creating a migration package:
+
+```bash
+mainsequence migrations scaffold \
+  --package msm \
+  --module migrations \
+  --namespace mainsequence.examples \
+  --base msm.base:MarketsBase \
+  --metadata msm.base:MarketsBase.metadata
+```
+
+The generated provider should use:
+
+- `build_alembic_version_metatable(...)`
+- `build_metatable_model_registry(...)`
+- `build_metatable_migration_provider(...)`
+- SDK-owned `run_mainsequence_alembic_env(...)`
+- SDK-owned `script.py.mako`
+
+Then use the normal lifecycle:
+
+```bash
+mainsequence migrations current --provider migrations:migration
+mainsequence migrations revision --provider migrations:migration
+mainsequence migrations upgrade --provider migrations:migration head
+```
+
+`revision` writes normal Alembic files. `upgrade` reserves provider MetaTables,
+runs Alembic DDL through the backend-issued migration credential, finalizes
+provider-scoped MetaTable catalog rows, and runs the optional provider hook.
+
+## Rules
+
+- Keep Alembic as the schema migration engine; do not build custom operation
+  lists or fake migration payload formats.
+- Keep provider scope explicit. Do not scan all imported models or installed
+  packages.
+- For one-model or configured dynamic providers, use `metadata_for_models(...)`
+  and a provider-specific `metatable_models` list.
+- Never send or thread request-side `data_source_uid` through migration status
+  or apply flows. Backend migration operations resolve the data source from the
+  registered Alembic version MetaTable UID.
+- Do not create SDK reset/reconcile commands for stale reserved state. If stale
+  reserved state exists, fail clearly and require an explicit backend/admin
+  repair path.
+- Do not write direct backend migration request bodies in examples. Use the CLI
+  and SDK provider APIs.
+- Do not call platform-managed model `.register()` in normal application code.
+  Registration is reserved for the migration workflow.
+
+## Debugging
+
+- If `current` fails before Alembic runs, inspect the provider import path and
+  Alembic version MetaTable binding.
+- If `revision` autogenerate tries to create everything again, the local
+  migration connection cannot see the provider's current physical tables.
+- If `upgrade` fails during prepare, inspect provider model identifiers,
+  physical table names, and reserved/active MetaTable rows.
+- If `upgrade` fails during finalization, inspect whether Alembic created the
+  physical tables for every provider-scoped model.
+- If an after-register hook reports the wrong model count, ensure it uses the
+  provider-scoped context, not a package-global model registry.

{mainsequence-4.3.8 → mainsequence-4.3.14}/agent_scaffold/skills/data_publishing/meta_tables/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: mainsequence-meta-tables
-description: Use this skill when the task is about defining, registering, querying, migrating, or reviewing Main Sequence MetaTables. This skill owns SQLAlchemy table contracts, backend-managed table registration, external table registration, Alembic-based MetaTable migrations, governed compiled SQL operations, foreign keys, indexes, and validation rules. It does not own DataNode producers, API route contracts, scheduling, releases, or sharing policy.
+description: Use this skill when the task is about defining, querying, or reviewing Main Sequence MetaTables. This skill owns SQLAlchemy table contracts, backend-managed table authoring, external table registration, governed compiled SQL operations, foreign keys, indexes, naming, cadence, and validation rules. For Alembic migration lifecycle work, use the mainsequence-metatable-migrations skill. It does not own DataNode producers, API route contracts, scheduling, releases, or sharing policy.
 ---
 
 # Main Sequence MetaTables
@@ -15,12 +15,11 @@ This skill is for schema-driven application tables registered through TS Manager
 
 - define SQLAlchemy/Core or ORM table models for `MetaTable` registration
 - choose `platform_managed` or `external_registered` management mode
-- register platform-managed tables through the model class API
+- declare platform-managed tables for migration-managed registration
 - build registration requests from resolved SQLAlchemy metadata when inspection is useful
 - define indexes and foreign keys in SQLAlchemy metadata for Alembic-owned DDL
 - design governed compiled SQL read and write operations
-- design provider-based Alembic contract evolution for MetaTables
-- run the documented `mainsequence migrations ...` lifecycle for Alembic-backed MetaTable changes
+- route provider-based Alembic contract evolution to the MetaTable migration skill
 - review table contracts for physical-name, namespace, and identifier issues
 - review whether a task should be a `MetaTable` or a `DataNode`
 
@@ -41,6 +40,8 @@ If the user is still in the discovery process and does not yet know what data ex
 
 - discovery-only data inventory before table implementation:
   `.agents/skills/mainsequence/data_access/exploration/SKILL.md`
+- MetaTable migrations:
+  `.agents/skills/mainsequence/data_publishing/meta_table_migrations/SKILL.md`
 - DataNodes:
   `.agents/skills/mainsequence/data_publishing/data_nodes/SKILL.md`
 - APIs and FastAPI:
@@ -55,7 +56,9 @@ If the user is still in the discovery process and does not yet know what data ex
 ## Read First
 
 1. `docs/tutorial/working_with_meta_tables.md`
-2. For migration work, `docs/tutorial/metatable_migrations.md`
+2. For migration work, use
+   `.agents/skills/mainsequence/data_publishing/meta_table_migrations/SKILL.md`
+   and `docs/tutorial/metatable_migrations.md`
 3. `docs/knowledge/meta_tables/index.md`
 4. `docs/knowledge/meta_tables/sqlalchemy.md`
 5. `docs/knowledge/meta_tables/compiled_sql.md`
@@ -111,9 +114,9 @@ creation or deletion.
 The only migration workflow to recommend is the Main Sequence CLI lifecycle:
 
 ```bash
-mainsequence migrations current --provider mainsequence_migrations:migration
-mainsequence migrations revision --provider mainsequence_migrations:migration
-mainsequence migrations upgrade --provider mainsequence_migrations:migration head
+mainsequence migrations current --provider sdk_examples.migrations:migration
+mainsequence migrations revision --provider sdk_examples.migrations:migration
+mainsequence migrations upgrade --provider sdk_examples.migrations:migration head
 ```
 
 ### 1. SQLAlchemy metadata is the authoring source
@@ -127,10 +130,22 @@ Do not hand-build contract fragments when the SQLAlchemy helper can derive them.
 For `platform_managed`, inherit from `PlatformManagedMetaTable`.
 
 Declare an explicit project-prefixed SQLAlchemy `__tablename__`. Use
-`schema_table_name(project_or_app, concept)` from `mainsequence.meta_tables` to
-generate that name. The mixin derives only the logical `storage_hash` from
-storage-relevant configuration and table shape; it must not use that hash as the
-SQLAlchemy table name.
+`schema_table_name(app, concept, suffix=None)` from `mainsequence.meta_tables`
+to generate that name:
+
+```python
+def schema_table_name(
+    app: str,
+    concept: str,
+    suffix: str | None = None,
+) -> str: ...
+```
+
+Use `app` for the project/package prefix, `concept` for the table concept, and
+`suffix` for a namespace, variant, or bounded specialization when the same
+concept exists in multiple logical scopes. The mixin derives only the logical
+`storage_hash` from storage-relevant configuration and table shape; it must not
+use that hash as the SQLAlchemy table name.
 
 When a platform-managed table must support in-place contract migrations from its
 first version, use Alembic. Keep the SDK model as a normal
@@ -179,6 +194,7 @@ from mainsequence.meta_tables import PlatformManagedMetaTable, schema_table_name
 
 PROJECT_NAME = "sdk_examples"
 ACCOUNT_TABLE_NAME = schema_table_name(PROJECT_NAME, "account")
+BROKER_ACCOUNT_TABLE_NAME = schema_table_name(PROJECT_NAME, "account", suffix="broker")
 
 
 class Account(PlatformManagedMetaTable, Base):
@@ -266,7 +282,7 @@ both the schema and the table's intention.
 Provider scope:
 
 ```python
-migration = AlembicMetaTableMigration(
+migration = build_metatable_migration_provider(
     ...,
     metatable_models=[Account, Asset],
 )
@@ -291,9 +307,10 @@ asset_meta_table = MetaTable.register(asset_request)
 
 ### 4. Schema changes use Alembic
 
-When doing migration work, first read
-`docs/tutorial/metatable_migrations.md`. That document is the tutorial source
-for the provider-based Alembic lifecycle.
+When doing migration work, use
+`.agents/skills/mainsequence/data_publishing/meta_table_migrations/SKILL.md`
+and read `docs/tutorial/metatable_migrations.md`. The migration skill owns the
+provider-based Alembic lifecycle.
 
 Do not apply in-place contract changes by changing a `PlatformManagedMetaTable`
 SQLAlchemy class and calling normal registration again. Shape-addressed
@@ -310,8 +327,10 @@ create a new Alembic revision on top of the current head.
 For contract evolution, define or update one selected
 `AlembicMetaTableMigration` provider:
 
-- put the provider in `mainsequence_migrations.py:migration` or pass
-  `--provider module.path:migration`
+- create or update a scaffolded package provider with
+  `mainsequence migrations scaffold`
+- pass the selected provider explicitly, for example
+  `--provider sdk_examples.migrations:migration`
 - set `package`, `migration_namespace`, `script_location`, and `target_metadata`
 - set `alembic_registry` to an `AlembicVersionMetaTable` subclass
 - list the post-apply catalog scope in `metatable_models`
@@ -326,17 +345,18 @@ SQLAlchemy table name used by the provider model. Keep that table name stable
 when a class is renamed or moved but must keep the same platform identity.
 When declaring an explicit identifier, explicit physical table name, or Alembic
 version table name, prefix it with the project or package name rather than using
-a bare table name. Use `schema_table_name(project_or_app, concept)` for the
-physical table and Alembic version table names.
+a bare table name. Use `schema_table_name(app, concept, suffix=None)` for the
+physical table and Alembic version table names. Use `suffix` for a namespace or
+variant, for example `schema_table_name("msm", "positions", suffix="broker")`.
 
-Do not ask users to construct backend migration payloads, call low-level
-migration request models, or use SDK helper functions directly. The backend
-request shape is reference material in the tutorial; the user-facing path is:
+Do not ask users to construct backend migration payloads or call low-level
+migration request models. The backend request shape is reference material in
+the tutorial; the user-facing path is:
 
 ```bash
-mainsequence migrations current --provider mainsequence_migrations:migration
-mainsequence migrations revision --provider mainsequence_migrations:migration
-mainsequence migrations upgrade --provider mainsequence_migrations:migration head
+mainsequence migrations current --provider sdk_examples.migrations:migration
+mainsequence migrations revision --provider sdk_examples.migrations:migration
+mainsequence migrations upgrade --provider sdk_examples.migrations:migration head
 ```
 
 All migration commands prepare the provider, reserve provider-scoped

{mainsequence-4.3.8 → mainsequence-4.3.14}/mainsequence/cli/migrations.py

@@ -3,6 +3,7 @@ from __future__ import annotations
 import dataclasses
 import json
 import logging
+import pathlib
 import re
 import sys
 from collections.abc import Mapping, Sequence
@@ -23,6 +24,7 @@ from mainsequence.meta_tables.migrations import (
     alembic_config_for_provider,
     apply_mainsequence_migration_role,
     load_alembic_metatable_migration_provider,
+    scaffold_migration_package,
 )
 
 migrations = typer.Typer(help="Alembic-owned MetaTable migration commands")
@@ -30,7 +32,8 @@ REGISTER_ENDPOINT = "/orm/api/ts_manager/meta_table/register/"
 METATABLE_COLLECTION_ENDPOINT = "/orm/api/ts_manager/meta_table/"
 DYNAMIC_TABLE_COLLECTION_ENDPOINT = "/orm/api/ts_manager/dynamic_table/"
 FINALIZE_MANAGED_ENDPOINT = "/orm/api/ts_manager/meta_table/finalize-managed/"
-ALEMBIC_PROVIDER_RESET_ENDPOINT = "/orm/api/ts_manager/meta_table/alembic-provider-reset/"
+DEFAULT_SCAFFOLD_PROJECT_ROOT = pathlib.Path(".")
+DEFAULT_SCAFFOLD_SOURCE_ROOT = pathlib.Path("src")
 
 
 class _AlembicOutput:
@@ -412,9 +415,22 @@ def _emit_metatable_reservation(model: type[Any], item: Any) -> None:
     )
 
 
+def _finalization_item_failed(item: Any) -> bool:
+    return (
+        _item_value(item, "provisioning_status") != "active"
+        or _item_value(item, "physical_table_exists") is False
+        or _item_value(item, "error") not in (None, "", {})
+    )
+
+
 def _emit_metatable_finalization(model: type[Any], item: Any) -> None:
     finalized = _item_value(item, "finalized")
-    action = "finalized" if finalized is not False else "finalize-failed"
+    if _finalization_item_failed(item):
+        action = "finalize-failed"
+    elif finalized is False:
+        action = "active"
+    else:
+        action = "finalized"
     _emit_progress(
         _metatable_message(
             endpoint=FINALIZE_MANAGED_ENDPOINT,
@@ -666,6 +682,113 @@ def _next_sequential_revision_id(
     return next_revision_id
 
 
+@migrations.command("scaffold")
+def scaffold(
+    package: str = typer.Option(
+        ...,
+        "--package",
+        help="Project package or migration provider package name, for example msm.",
+    ),
+    namespace: str = typer.Option(
+        ...,
+        "--namespace",
+        help="Migration namespace for this provider.",
+    ),
+    metadata: str = typer.Option(
+        ...,
+        "--metadata",
+        help="Target SQLAlchemy metadata reference in module:object form.",
+    ),
+    module: str = typer.Option(
+        "migrations",
+        "--module",
+        help="Python module to create, for example migrations or msm.migrations.",
+    ),
+    project_root: pathlib.Path = typer.Option(  # noqa: B008
+        DEFAULT_SCAFFOLD_PROJECT_ROOT,
+        "--project-root",
+        help="Project root where the source tree lives.",
+    ),
+    source_root: pathlib.Path = typer.Option(  # noqa: B008
+        DEFAULT_SCAFFOLD_SOURCE_ROOT,
+        "--source-root",
+        help="Source root under project root.",
+    ),
+    base: str | None = typer.Option(
+        None,
+        "--base",
+        help="Optional project declarative base reference in module:object form.",
+    ),
+    models: str | None = typer.Option(
+        None,
+        "--models",
+        help="Optional model registry function reference in module:object form.",
+    ),
+    alembic_version_name: str = typer.Option(
+        "ProjectAlembicVersion",
+        "--alembic-version-name",
+        help="Generated Alembic version MetaTable class name.",
+    ),
+    alembic_version_identifier: str | None = typer.Option(
+        None,
+        "--alembic-version-identifier",
+        help="Generated Alembic version MetaTable identifier.",
+    ),
+    alembic_version_schema: str | None = typer.Option(
+        "public",
+        "--alembic-version-schema",
+        help="Physical schema for the Alembic version table. Pass an empty string for no schema.",
+    ),
+    alembic_version_table_name: str = typer.Option(
+        "alembic_version",
+        "--alembic-version-table-name",
+        help="Physical Alembic version table name.",
+    ),
+    force: bool = typer.Option(
+        False,
+        "--force",
+        help="Overwrite changed scaffold files.",
+    ),
+    json_output: bool = typer.Option(False, "--json", help="Emit JSON."),
+) -> None:
+    """Create an SDK-shaped MetaTable migration package skeleton."""
+
+    normalized_schema = alembic_version_schema
+    if normalized_schema == "":
+        normalized_schema = None
+    try:
+        result = scaffold_migration_package(
+            project_root=project_root,
+            module=module,
+            package=package,
+            namespace=namespace,
+            metadata_ref=metadata,
+            base_ref=base,
+            model_registry_ref=models,
+            alembic_version_name=alembic_version_name,
+            alembic_version_identifier=alembic_version_identifier,
+            alembic_version_schema=normalized_schema,
+            alembic_version_table_name=alembic_version_table_name,
+            source_root=source_root,
+            force=force,
+        )
+    except (FileExistsError, ValueError) as exc:
+        raise typer.BadParameter(str(exc)) from exc
+
+    for file in result.files:
+        _emit_status(f"Scaffold {file.action} {file.path}")
+    _emit(
+        {
+            "root": str(result.root),
+            "files": [
+                {"path": str(file.path), "action": file.action}
+                for file in result.files
+            ],
+        },
+        json_output=json_output,
+    )
+
+
 @migrations.command("current")
 def current(
     provider: str | None = typer.Option(
@@ -884,57 +1007,3 @@ def downgrade(
         },
         json_output=json_output,
     )
-
-
-@migrations.command("reset")
-def reset(
-    provider: str | None = typer.Option(
-        None,
-        "--provider",
-        help="Migration provider reference, for example msm.migrations:migration.",
-    ),
-    confirm_reset: bool = typer.Option(
-        False,
-        "--confirm-reset",
-        help="Required confirmation for destructive provider-scoped reset.",
-    ),
-    drop_physical_tables: bool = typer.Option(
-        True,
-        "--drop-physical-tables/--keep-physical-tables",
-        help="Drop provider physical tables during reset.",
-    ),
-    clear_alembic_version_table: bool = typer.Option(
-        True,
-        "--clear-alembic-version-table/--keep-alembic-version-table",
-        help="Clear the provider Alembic version table during reset.",
-    ),
-    include_reserved: bool = typer.Option(
-        True,
-        "--include-reserved/--active-only",
-        help="Include already-reserved provider MetaTables in reset results.",
-    ),
-    timeout: float | None = typer.Option(None, "--timeout"),
-    json_output: bool = typer.Option(False, "--json", help="Emit JSON."),
-) -> None:
-    """Reset an Alembic-managed provider catalog/physical state."""
-
-    if not confirm_reset:
-        raise typer.BadParameter(
-            "Pass --confirm-reset to call the destructive provider reset endpoint.",
-            param_hint="--confirm-reset",
-        )
-    migration = _load_migration(provider)
-    _emit_status(
-        "Calling provider reset endpoint "
-        f"{ALEMBIC_PROVIDER_RESET_ENDPOINT} provider={migration.migration_provider_key}..."
-    )
-    response = migration.reset_alembic_provider(
-        confirm_reset=True,
-        drop_physical_tables=drop_physical_tables,
-        clear_alembic_version_table=clear_alembic_version_table,
-        include_reserved=include_reserved,
-        timeout=timeout,
-        on_reset_status=_emit_status,
-    )
-    _emit_status("Alembic provider reset finished.")
-    _emit(response, json_output=json_output)