mainsequence 4.3.9__tar.gz → 4.3.16__tar.gz

{mainsequence-4.3.9/mainsequence.egg-info → mainsequence-4.3.16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence
-Version: 4.3.9
+Version: 4.3.16
 Summary: Main Sequence SDK 
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License: MainSequence GmbH SDK License Agreement

mainsequence-4.3.16/agent_scaffold/skills/data_publishing/meta_table_migrations/SKILL.md

@@ -0,0 +1,102 @@
+---
+name: mainsequence-metatable-migrations
+description: Use this skill when the task is about Main Sequence MetaTable schema evolution, Alembic provider setup, migration scaffold generation, revision creation, migration upgrade/current commands, provider-scoped MetaTable reservation/finalization, or debugging migration lifecycle failures. This skill owns the migration-first workflow for platform-managed MetaTables and should be used instead of direct MetaTable.register() guidance.
+---
+
+# Main Sequence MetaTable Migrations
+
+## Overview
+
+Use this skill for platform-managed MetaTable schema lifecycle work. The
+standard path is migration-first: declare SQLAlchemy MetaTable models, put them
+in a provider, run `mainsequence migrations ...`, and let the SDK reserve,
+migrate, finalize, and bind provider-scoped MetaTables.
+
+Do not recommend direct `MetaTable.register()` or model `.register()` calls for
+platform-managed application tables outside the migration workflow.
+
+## Read First
+
+1. `docs/tutorial/metatable_migrations.md`
+2. `docs/knowledge/meta_tables/migrations.md`
+3. `docs/knowledge/meta_tables/api.md`
+4. `docs/adr/0020-metatable-migration-artifact-registry.md`
+5. `docs/adr/0026-sdk-owned-migration-scaffolding.md`
+
+## Required Decisions
+
+- Which provider module owns the migration stream?
+- Which package and migration namespace identify the provider?
+- Which `AlembicVersionMetaTable` binding points to Alembic's version table?
+- Which SQLAlchemy `MetaData` object is the target metadata?
+- Which provider-scoped MetaTable models belong in `metatable_models`?
+- Does a dynamic provider need `metadata_for_models(...)` instead of full
+  package metadata?
+- Does the project need an `after_register_metatables` catalog hook, and does
+  that hook use `context.metatable_models` and `context.registered_metatables`
+  instead of importing a broader registry?
+
+## Standard Workflow
+
+Use SDK-owned scaffold and helpers when creating a migration package:
+
+```bash
+mainsequence migrations scaffold \
+  --package msm \
+  --module migrations \
+  --namespace mainsequence.examples \
+  --base msm.base:MarketsBase \
+  --metadata msm.base:MarketsBase.metadata
+```
+
+The generated provider should use:
+
+- `build_alembic_version_metatable(...)`
+- `build_metatable_model_registry(...)`
+- `build_metatable_migration_provider(...)`
+- SDK-owned `run_mainsequence_alembic_env(...)`
+- SDK-owned `script.py.mako`
+
+Then use the normal lifecycle:
+
+```bash
+mainsequence migrations current --provider migrations:migration
+mainsequence migrations revision --provider migrations:migration
+mainsequence migrations upgrade --provider migrations:migration head
+```
+
+`revision` writes normal Alembic files. `upgrade` reserves provider MetaTables,
+runs Alembic DDL through the backend-issued migration credential, finalizes
+provider-scoped MetaTable catalog rows, and runs the optional provider hook.
+
+## Rules
+
+- Keep Alembic as the schema migration engine; do not build custom operation
+  lists or fake migration payload formats.
+- Keep provider scope explicit. Do not scan all imported models or installed
+  packages.
+- For one-model or configured dynamic providers, use `metadata_for_models(...)`
+  and a provider-specific `metatable_models` list.
+- Never send or thread request-side `data_source_uid` through migration status
+  or apply flows. Backend migration operations resolve the data source from the
+  registered Alembic version MetaTable UID.
+- Do not create SDK reset/reconcile commands for stale reserved state. If stale
+  reserved state exists, fail clearly and require an explicit backend/admin
+  repair path.
+- Do not write direct backend migration request bodies in examples. Use the CLI
+  and SDK provider APIs.
+- Do not call platform-managed model `.register()` in normal application code.
+  Registration is reserved for the migration workflow.
+
+## Debugging
+
+- If `current` fails before Alembic runs, inspect the provider import path and
+  Alembic version MetaTable binding.
+- If `revision` autogenerate tries to create everything again, the local
+  migration connection cannot see the provider's current physical tables.
+- If `upgrade` fails during prepare, inspect provider model identifiers,
+  physical table names, and reserved/active MetaTable rows.
+- If `upgrade` fails during finalization, inspect whether Alembic created the
+  physical tables for every provider-scoped model.
+- If an after-register hook reports the wrong model count, ensure it uses the
+  provider-scoped context, not a package-global model registry.

{mainsequence-4.3.9 → mainsequence-4.3.16}/agent_scaffold/skills/data_publishing/meta_tables/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: mainsequence-meta-tables
-description: Use this skill when the task is about defining, registering, querying, migrating, or reviewing Main Sequence MetaTables. This skill owns SQLAlchemy table contracts, backend-managed table registration, external table registration, Alembic-based MetaTable migrations, governed compiled SQL operations, foreign keys, indexes, and validation rules. It does not own DataNode producers, API route contracts, scheduling, releases, or sharing policy.
+description: Use this skill when the task is about defining, querying, or reviewing Main Sequence MetaTables. This skill owns SQLAlchemy table contracts, backend-managed table authoring, external table registration, governed compiled SQL operations, foreign keys, indexes, naming, cadence, and validation rules. For Alembic migration lifecycle work, use the mainsequence-metatable-migrations skill. It does not own DataNode producers, API route contracts, scheduling, releases, or sharing policy.
 ---
 
 # Main Sequence MetaTables
@@ -15,12 +15,11 @@ This skill is for schema-driven application tables registered through TS Manager
 
 - define SQLAlchemy/Core or ORM table models for `MetaTable` registration
 - choose `platform_managed` or `external_registered` management mode
-- register platform-managed tables through the model class API
+- declare platform-managed tables for migration-managed registration
 - build registration requests from resolved SQLAlchemy metadata when inspection is useful
 - define indexes and foreign keys in SQLAlchemy metadata for Alembic-owned DDL
 - design governed compiled SQL read and write operations
-- design provider-based Alembic contract evolution for MetaTables
-- run the documented `mainsequence migrations ...` lifecycle for Alembic-backed MetaTable changes
+- route provider-based Alembic contract evolution to the MetaTable migration skill
 - review table contracts for physical-name, namespace, and identifier issues
 - review whether a task should be a `MetaTable` or a `DataNode`
 
@@ -41,6 +40,8 @@ If the user is still in the discovery process and does not yet know what data ex
 
 - discovery-only data inventory before table implementation:
   `.agents/skills/mainsequence/data_access/exploration/SKILL.md`
+- MetaTable migrations:
+  `.agents/skills/mainsequence/data_publishing/meta_table_migrations/SKILL.md`
 - DataNodes:
   `.agents/skills/mainsequence/data_publishing/data_nodes/SKILL.md`
 - APIs and FastAPI:
@@ -55,7 +56,9 @@ If the user is still in the discovery process and does not yet know what data ex
 ## Read First
 
 1. `docs/tutorial/working_with_meta_tables.md`
-2. For migration work, `docs/tutorial/metatable_migrations.md`
+2. For migration work, use
+   `.agents/skills/mainsequence/data_publishing/meta_table_migrations/SKILL.md`
+   and `docs/tutorial/metatable_migrations.md`
 3. `docs/knowledge/meta_tables/index.md`
 4. `docs/knowledge/meta_tables/sqlalchemy.md`
 5. `docs/knowledge/meta_tables/compiled_sql.md`
@@ -111,9 +114,9 @@ creation or deletion.
 The only migration workflow to recommend is the Main Sequence CLI lifecycle:
 
 ```bash
-mainsequence migrations current --provider mainsequence_migrations:migration
-mainsequence migrations revision --provider mainsequence_migrations:migration
-mainsequence migrations upgrade --provider mainsequence_migrations:migration head
+mainsequence migrations current --provider sdk_examples.migrations:migration
+mainsequence migrations revision --provider sdk_examples.migrations:migration
+mainsequence migrations upgrade --provider sdk_examples.migrations:migration head
 ```
 
 ### 1. SQLAlchemy metadata is the authoring source
@@ -279,7 +282,7 @@ both the schema and the table's intention.
 Provider scope:
 
 ```python
-migration = AlembicMetaTableMigration(
+migration = build_metatable_migration_provider(
     ...,
     metatable_models=[Account, Asset],
 )
@@ -304,9 +307,10 @@ asset_meta_table = MetaTable.register(asset_request)
 
 ### 4. Schema changes use Alembic
 
-When doing migration work, first read
-`docs/tutorial/metatable_migrations.md`. That document is the tutorial source
-for the provider-based Alembic lifecycle.
+When doing migration work, use
+`.agents/skills/mainsequence/data_publishing/meta_table_migrations/SKILL.md`
+and read `docs/tutorial/metatable_migrations.md`. The migration skill owns the
+provider-based Alembic lifecycle.
 
 Do not apply in-place contract changes by changing a `PlatformManagedMetaTable`
 SQLAlchemy class and calling normal registration again. Shape-addressed
@@ -323,8 +327,10 @@ create a new Alembic revision on top of the current head.
 For contract evolution, define or update one selected
 `AlembicMetaTableMigration` provider:
 
-- put the provider in `mainsequence_migrations.py:migration` or pass
-  `--provider module.path:migration`
+- create or update a scaffolded package provider with
+  `mainsequence migrations scaffold`
+- pass the selected provider explicitly, for example
+  `--provider sdk_examples.migrations:migration`
 - set `package`, `migration_namespace`, `script_location`, and `target_metadata`
 - set `alembic_registry` to an `AlembicVersionMetaTable` subclass
 - list the post-apply catalog scope in `metatable_models`
@@ -343,14 +349,14 @@ a bare table name. Use `schema_table_name(app, concept, suffix=None)` for the
 physical table and Alembic version table names. Use `suffix` for a namespace or
 variant, for example `schema_table_name("msm", "positions", suffix="broker")`.
 
-Do not ask users to construct backend migration payloads, call low-level
-migration request models, or use SDK helper functions directly. The backend
-request shape is reference material in the tutorial; the user-facing path is:
+Do not ask users to construct backend migration payloads or call low-level
+migration request models. The backend request shape is reference material in
+the tutorial; the user-facing path is:
 
 ```bash
-mainsequence migrations current --provider mainsequence_migrations:migration
-mainsequence migrations revision --provider mainsequence_migrations:migration
-mainsequence migrations upgrade --provider mainsequence_migrations:migration head
+mainsequence migrations current --provider sdk_examples.migrations:migration
+mainsequence migrations revision --provider sdk_examples.migrations:migration
+mainsequence migrations upgrade --provider sdk_examples.migrations:migration head
 ```
 
 All migration commands prepare the provider, reserve provider-scoped

{mainsequence-4.3.9 → mainsequence-4.3.16}/mainsequence/cli/migrations.py

@@ -3,6 +3,7 @@ from __future__ import annotations
 import dataclasses
 import json
 import logging
+import pathlib
 import re
 import sys
 from collections.abc import Mapping, Sequence
@@ -23,6 +24,7 @@ from mainsequence.meta_tables.migrations import (
     alembic_config_for_provider,
     apply_mainsequence_migration_role,
     load_alembic_metatable_migration_provider,
+    scaffold_migration_package,
 )
 
 migrations = typer.Typer(help="Alembic-owned MetaTable migration commands")
@@ -30,7 +32,8 @@ REGISTER_ENDPOINT = "/orm/api/ts_manager/meta_table/register/"
 METATABLE_COLLECTION_ENDPOINT = "/orm/api/ts_manager/meta_table/"
 DYNAMIC_TABLE_COLLECTION_ENDPOINT = "/orm/api/ts_manager/dynamic_table/"
 FINALIZE_MANAGED_ENDPOINT = "/orm/api/ts_manager/meta_table/finalize-managed/"
-ALEMBIC_PROVIDER_RESET_ENDPOINT = "/orm/api/ts_manager/meta_table/alembic-provider-reset/"
+DEFAULT_SCAFFOLD_PROJECT_ROOT = pathlib.Path(".")
+DEFAULT_SCAFFOLD_SOURCE_ROOT = pathlib.Path("src")
 
 
 class _AlembicOutput:
@@ -412,9 +415,22 @@ def _emit_metatable_reservation(model: type[Any], item: Any) -> None:
     )
 
 
+def _finalization_item_failed(item: Any) -> bool:
+    return (
+        _item_value(item, "provisioning_status") != "active"
+        or _item_value(item, "physical_table_exists") is False
+        or _item_value(item, "error") not in (None, "", {})
+    )
+
+
 def _emit_metatable_finalization(model: type[Any], item: Any) -> None:
     finalized = _item_value(item, "finalized")
-    action = "finalized" if finalized is not False else "finalize-failed"
+    if _finalization_item_failed(item):
+        action = "finalize-failed"
+    elif finalized is False:
+        action = "active"
+    else:
+        action = "finalized"
     _emit_progress(
         _metatable_message(
             endpoint=FINALIZE_MANAGED_ENDPOINT,
@@ -666,6 +682,113 @@ def _next_sequential_revision_id(
     return next_revision_id
 
 
+@migrations.command("scaffold")
+def scaffold(
+    package: str = typer.Option(
+        ...,
+        "--package",
+        help="Project package or migration provider package name, for example msm.",
+    ),
+    namespace: str = typer.Option(
+        ...,
+        "--namespace",
+        help="Migration namespace for this provider.",
+    ),
+    metadata: str = typer.Option(
+        ...,
+        "--metadata",
+        help="Target SQLAlchemy metadata reference in module:object form.",
+    ),
+    module: str = typer.Option(
+        "migrations",
+        "--module",
+        help="Python module to create, for example migrations or msm.migrations.",
+    ),
+    project_root: pathlib.Path = typer.Option(  # noqa: B008
+        DEFAULT_SCAFFOLD_PROJECT_ROOT,
+        "--project-root",
+        help="Project root where the source tree lives.",
+    ),
+    source_root: pathlib.Path = typer.Option(  # noqa: B008
+        DEFAULT_SCAFFOLD_SOURCE_ROOT,
+        "--source-root",
+        help="Source root under project root.",
+    ),
+    base: str | None = typer.Option(
+        None,
+        "--base",
+        help="Optional project declarative base reference in module:object form.",
+    ),
+    models: str | None = typer.Option(
+        None,
+        "--models",
+        help="Optional model registry function reference in module:object form.",
+    ),
+    alembic_version_name: str = typer.Option(
+        "ProjectAlembicVersion",
+        "--alembic-version-name",
+        help="Generated Alembic version MetaTable class name.",
+    ),
+    alembic_version_identifier: str | None = typer.Option(
+        None,
+        "--alembic-version-identifier",
+        help="Generated Alembic version MetaTable identifier.",
+    ),
+    alembic_version_schema: str | None = typer.Option(
+        "public",
+        "--alembic-version-schema",
+        help="Physical schema for the Alembic version table. Pass an empty string for no schema.",
+    ),
+    alembic_version_table_name: str = typer.Option(
+        "alembic_version",
+        "--alembic-version-table-name",
+        help="Physical Alembic version table name.",
+    ),
+    force: bool = typer.Option(
+        False,
+        "--force",
+        help="Overwrite changed scaffold files.",
+    ),
+    json_output: bool = typer.Option(False, "--json", help="Emit JSON."),
+) -> None:
+    """Create an SDK-shaped MetaTable migration package skeleton."""
+
+    normalized_schema = alembic_version_schema
+    if normalized_schema == "":
+        normalized_schema = None
+    try:
+        result = scaffold_migration_package(
+            project_root=project_root,
+            module=module,
+            package=package,
+            namespace=namespace,
+            metadata_ref=metadata,
+            base_ref=base,
+            model_registry_ref=models,
+            alembic_version_name=alembic_version_name,
+            alembic_version_identifier=alembic_version_identifier,
+            alembic_version_schema=normalized_schema,
+            alembic_version_table_name=alembic_version_table_name,
+            source_root=source_root,
+            force=force,
+        )
+    except (FileExistsError, ValueError) as exc:
+        raise typer.BadParameter(str(exc)) from exc
+
+    for file in result.files:
+        _emit_status(f"Scaffold {file.action} {file.path}")
+    _emit(
+        {
+            "root": str(result.root),
+            "files": [
+                {"path": str(file.path), "action": file.action}
+                for file in result.files
+            ],
+        },
+        json_output=json_output,
+    )
+
+
 @migrations.command("current")
 def current(
     provider: str | None = typer.Option(
@@ -884,57 +1007,3 @@ def downgrade(
         },
         json_output=json_output,
     )
-
-
-@migrations.command("reset")
-def reset(
-    provider: str | None = typer.Option(
-        None,
-        "--provider",
-        help="Migration provider reference, for example msm.migrations:migration.",
-    ),
-    confirm_reset: bool = typer.Option(
-        False,
-        "--confirm-reset",
-        help="Required confirmation for destructive provider-scoped reset.",
-    ),
-    drop_physical_tables: bool = typer.Option(
-        True,
-        "--drop-physical-tables/--keep-physical-tables",
-        help="Drop provider physical tables during reset.",
-    ),
-    clear_alembic_version_table: bool = typer.Option(
-        True,
-        "--clear-alembic-version-table/--keep-alembic-version-table",
-        help="Clear the provider Alembic version table during reset.",
-    ),
-    include_reserved: bool = typer.Option(
-        True,
-        "--include-reserved/--active-only",
-        help="Include already-reserved provider MetaTables in reset results.",
-    ),
-    timeout: float | None = typer.Option(None, "--timeout"),
-    json_output: bool = typer.Option(False, "--json", help="Emit JSON."),
-) -> None:
-    """Reset an Alembic-managed provider catalog/physical state."""
-
-    if not confirm_reset:
-        raise typer.BadParameter(
-            "Pass --confirm-reset to call the destructive provider reset endpoint.",
-            param_hint="--confirm-reset",
-        )
-    migration = _load_migration(provider)
-    _emit_status(
-        "Calling provider reset endpoint "
-        f"{ALEMBIC_PROVIDER_RESET_ENDPOINT} provider={migration.migration_provider_key}..."
-    )
-    response = migration.reset_alembic_provider(
-        confirm_reset=True,
-        drop_physical_tables=drop_physical_tables,
-        clear_alembic_version_table=clear_alembic_version_table,
-        include_reserved=include_reserved,
-        timeout=timeout,
-        on_reset_status=_emit_status,
-    )
-    _emit_status("Alembic provider reset finished.")
-    _emit(response, json_output=json_output)