ms-markets 2.2.6.dev26__tar.gz → 2.2.6.dev27__tar.gz

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/CHANGELOG.md

@@ -7,6 +7,21 @@ and this project follows versioned releases.
 
 ## [Unreleased]
 
+### Changed
+
+- Requires `mainsequence-metatable>=0.1.31,<0.2`; the lock and runtime export
+  select 0.1.31, while Main Sequence remains 9.0.19. The shared migration runner
+  batches table inventories and inspections and waits up to 300 seconds for a
+  waking API, addressing the 30-second discovery timeout in the failed Markets
+  deployment ([MetaTables#55](https://github.com/mainsequence-projects/MetaTables/pull/55)).
+  Server-side batching, unchanged-contract write avoidance, and startup
+  improvements also require deployment of the updated shared MetaTables API.
+- The migration Job exposes the shared runner's INFO phase timings, always
+  reconciles the catalog even at head, and propagates failures to block rollout.
+  Provider identity, version-table binding, and applied revisions are unchanged.
+- Refreshed the MetaTables-owned skills and guide snapshots to 0.1.31; aligned
+  migration documentation, release guidance, tutorial, and print-only example.
+
 ## [2.2.6] - 2026-10-09
 
 ### Added

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ms-markets
-Version: 2.2.6.dev26
+Version: 2.2.6.dev27
 Summary: Financial markets ORM, engines, and application management extensions for Main Sequence.
 Project-URL: Changelog, https://github.com/mainsequence-projects/MainSequenceMarkets/blob/main/CHANGELOG.md
 Project-URL: Source, https://github.com/mainsequence-projects/MainSequenceMarkets
@@ -227,7 +227,7 @@ Requires-Python: <3.14,>=3.13
 Requires-Dist: alembic>=1.18.4
 Requires-Dist: cachetools
 Requires-Dist: fastapi>=0.136.3
-Requires-Dist: mainsequence-metatable<0.2,>=0.1.29
+Requires-Dist: mainsequence-metatable<0.2,>=0.1.31
 Requires-Dist: mainsequence<10,>=9.0.19
 Requires-Dist: numpy
 Requires-Dist: pandas
@@ -385,7 +385,7 @@ Recommended entry points:
 ## Quick Start
 
 `ms-markets` 2.x requires Python 3.13, Main Sequence SDK `>=9.0.19,<10`, and
-the MetaTables client `mainsequence-metatable>=0.1.29,<0.2`, which is imported
+the MetaTables client `mainsequence-metatable>=0.1.31,<0.2`, which is imported
 as `metatables`. SDK 8 and earlier are not supported; install a 1.x
 `ms-markets` release when maintaining an SDK 8 project. See
 [ADR 0043](https://mainsequence-projects.github.io/MainSequenceMarkets/ADR/0043-sdk-9-metatables-client-hard-cut/)
@@ -503,7 +503,7 @@ The core stack starts with:
 
 - `mainsequence>=9.0.19,<10` for identity, login, Git source context,
   CodeRepository, jobs, and agents
-- `mainsequence-metatable>=0.1.29,<0.2` (import `metatables`) for MetaTables,
+- `mainsequence-metatable>=0.1.31,<0.2` (import `metatables`) for MetaTables,
   time-index table updaters, governed compiled SQL, and application-owned
   Alembic migrations
 - `SQLAlchemy` for market-domain ORM models

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/README.md

@@ -108,7 +108,7 @@ Recommended entry points:
 ## Quick Start
 
 `ms-markets` 2.x requires Python 3.13, Main Sequence SDK `>=9.0.19,<10`, and
-the MetaTables client `mainsequence-metatable>=0.1.29,<0.2`, which is imported
+the MetaTables client `mainsequence-metatable>=0.1.31,<0.2`, which is imported
 as `metatables`. SDK 8 and earlier are not supported; install a 1.x
 `ms-markets` release when maintaining an SDK 8 project. See
 [ADR 0043](https://mainsequence-projects.github.io/MainSequenceMarkets/ADR/0043-sdk-9-metatables-client-hard-cut/)
@@ -226,7 +226,7 @@ The core stack starts with:
 
 - `mainsequence>=9.0.19,<10` for identity, login, Git source context,
   CodeRepository, jobs, and agents
-- `mainsequence-metatable>=0.1.29,<0.2` (import `metatables`) for MetaTables,
+- `mainsequence-metatable>=0.1.31,<0.2` (import `metatables`) for MetaTables,
   time-index table updaters, governed compiled SQL, and application-owned
   Alembic migrations
 - `SQLAlchemy` for market-domain ORM models

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/docs/fast_api/v1/index.md

@@ -173,11 +173,13 @@ push alone as deployment success.
 
 Runtime dependencies must be resolvable from the backend build environment.
 The published `ms-markets` 2.x package therefore declares
-`mainsequence>=9.0.19,<10` and `mainsequence-metatable>=0.1.29,<0.2` without
+`mainsequence>=9.0.19,<10` and `mainsequence-metatable>=0.1.31,<0.2` without
 exact patch pins. The lower bounds enforce the SDK 9 and MetaTables client
 extraction hard cut and the SDK request identity that the platform's current
 caller assertions require (SDK 9.0.18 and later), while the project lock and exported runtime requirements
-select the exact releases validated for this repository. Do not replace the
+select the exact releases validated for this repository. MetaTables 0.1.31 also
+provides batched migration inspections and a five-minute migration API discovery
+budget. Do not replace the
 published dependencies with machine-local `[tool.uv.sources]` path overrides.
 
 `create_app()` installs the SDK request identity with

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/docs/getting-started.md

@@ -1,7 +1,7 @@
 # Getting Started
 
 `ms-markets` 2.x requires Python 3.13, Main Sequence SDK `>=9.0.19,<10`, and
-the MetaTables client `mainsequence-metatable>=0.1.29,<0.2` (imported as
+the MetaTables client `mainsequence-metatable>=0.1.31,<0.2` (imported as
 `metatables`). It cannot run against an SDK 8 or earlier contract and does not
 provide compatibility aliases or a mixed-version mode. Install a 1.x
 `ms-markets` release when maintaining an SDK 8 project.

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/docs/knowledge/msm/migrations/index.md

@@ -60,6 +60,36 @@ This repository's `.mainsequence/workflows/ms-markets-api.yaml` builds the image
 at head is left unchanged, and a failure blocks the API rollout while the
 previous release keeps serving.
 
+### Migration diagnostics and repeated runs
+
+The minimum client is MetaTables 0.1.31. `upgrade_application()` automatically
+uses fresh table inventories and batched physical inspections; PostgreSQL
+reflection is batched by schema. Finalization requests remain bounded to ten
+tables each, with the version table first. No batching option or project-specific
+runner is required.
+
+Discovery during a migration waits up to 300 seconds for MetaTables to wake or
+finish replacing its pods. Ordinary client calls retain a 30-second discovery
+budget. A discovery failure still fails the Job and blocks deployment.
+
+`jobs/migrate_markets.py` enables INFO logging for `metatables.migrations.runner`.
+Its logs separate preparation, database setup, table checks, Alembic execution,
+catalog finalization, and total time. Measure these separately from platform
+scheduling and workflow coordination; do not log migration connection responses
+or credentials.
+
+An already-current revision **still runs catalog reconciliation**, including
+missing-table checks and description, label, and column metadata refresh. Do not
+skip the Job or finalization because `migrated=False`. If DDL committed but
+finalization failed, inspect the actual schema and per-table errors, then rerun
+the same migration through a new deployment. Do not stamp, rewrite applied
+revisions, or downgrade to recover catalog state.
+
+Client improvements require an updated application image. API-side batching,
+unchanged-contract write avoidance, and startup optimizations require the shared
+MetaTables API to be deployed with that release too. Update matching guidance
+with `metatables copy-metatables-skills --path .` after updating the client.
+
 !!! warning "Required: MetaTables access for the deployment"
     The migration Job runs as its own workload User, which has no MetaTables
     grants until an Organization admin runs `scripts/bootstrap_metatables_access.py`

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/docs/knowledge/msm/platform/metatable_migrations.md

@@ -110,7 +110,7 @@ history is the Alembic revision graph plus the provider's version table.
 
 ## Client Requirement
 
-The implementation requires `mainsequence-metatable>=0.1.29,<0.2` with
+The implementation requires `mainsequence-metatable>=0.1.31,<0.2` with
 `mainsequence>=9.0.19,<10`. The client exposes `AlembicMetaTableMigration`,
 `AlembicVersionMetaTable`, application-owned Alembic execution, and the command
 shape where
@@ -122,3 +122,11 @@ registry, version-table binding, revision IDs, and applied revision history.
 The 2.1.0 squash into `0018_initial_schema` is a separate, one-time replacement
 of the history; see
 [ADR 0045](../../../ADR/0045-squashed-initial-schema-and-namespaced-packages.md).
+
+The shared `metatables.upgrade_application(...)` lifecycle automatically batches
+table inventories and physical inspection. Migration API discovery waits up to
+300 seconds for a waking or restarting API; ordinary client calls retain their
+30-second budget. No private batching helper, timeout override, custom retry
+loop, or new revision is needed. An unchanged revision still finalizes catalog
+contracts and checks for missing tables. See
+[Migration diagnostics](../migrations/index.md#migration-diagnostics-and-repeated-runs).

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/docs/releasing.md

@@ -129,6 +129,27 @@ rebuilds the image, runs the `migrate-markets` Job and redeploys the API. Pushes
 `main` happen only through release merges, so the `main` deployment follows
 stable releases.
 
+### Upgrading the migration client
+
+The project requires `mainsequence-metatable>=0.1.31,<0.2`; `uv.lock` and
+`requirements.txt` select 0.1.31. After changing the dependency, run
+`mainsequence code-repository sync --path .`, refresh the MetaTables-owned
+guidance with `metatables copy-metatables-skills --path .`, and run the checks
+above. Keep the provider identity, version table, and applied revisions unchanged.
+
+The candidate image uses the shared `upgrade_application()` runner. Version
+0.1.31 adds batched inspections and a 300-second discovery budget for a waking
+API. The Job enables its INFO phase timings and must run even when Alembic is
+already at head: catalog reconciliation is not optional. The workflow remains
+image → migration Job → API deployment, all from the same candidate image.
+
+Deploy the shared MetaTables API as well to receive its server-side inspection,
+unchanged-contract, and startup improvements. A client-only upgrade cannot
+change an older deployed API. Verify image preparation, successful migration
+finalization, and API rollout separately; a push or an `already current` Alembic
+revision alone is not proof of deployment success. See
+[Migration diagnostics](knowledge/msm/migrations/index.md#migration-diagnostics-and-repeated-runs).
+
 ### MetaTables access in each Environment
 
 !!! warning "Required in every Environment"

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/docs/tutorial/index.md

@@ -15,6 +15,22 @@ Before starting, set up your environment with
 (typed `msm.api` row APIs, explicit MetaTable runtime attachment, and TimeIndexTableUpdater
 helpers for time-indexed facts).
 
+### Prepare the schema before starting runtime code
+
+1. Install the locked dependencies, including MetaTables 0.1.31 or later.
+2. For local ms-markets schema development, verify the local MetaTables runtime
+   and use `metatables --local migrations upgrade --provider msm_migrations:migration head`.
+   For a hosted environment, the ms-markets deployment's migration Job applies
+   this provider before the API rolls out; do not run it from your laptop.
+3. Start tutorial code with `msm.start_engine(...)` only after the tables have
+   been finalized. A consuming project uses the schema owned by the ms-markets
+   deployment and migrates only its own providers.
+
+Run `python examples/msm/platform/metatable_migration_lifecycle.py` for a safe,
+print-only walkthrough. The shared runner batches inspections automatically and
+reconciles the catalog even at the current revision. See
+[Migrations](../knowledge/msm/migrations/index.md) for the workflow and timing logs.
+
 ## Installing MS Markets Agent Skills
 
 Use the `msm` CLI when a host Main Sequence CodeRepository should receive the

ms_markets-2.2.6.dev27/examples/msm/platform/metatable_migration_lifecycle.py

@@ -0,0 +1,28 @@
+from __future__ import annotations
+
+PROVIDER = "msm_migrations:migration"
+
+
+def main() -> None:
+    print("ms-markets MetaTable migrations use the application-owned MetaTables provider:")
+    print(PROVIDER)
+
+    print("\nLocal schema-development command sequence (prints only; executes no DDL):")
+    for command in (
+        f"metatables --local --json migrations current --provider {PROVIDER}",
+        f'metatables --local migrations revision --provider {PROVIDER} -m "describe change"',
+        f"metatables --local migrations upgrade --provider {PROVIDER} head",
+        f"metatables --local migrations downgrade --provider {PROVIDER} <revision>",
+    ):
+        print(command)
+
+    print("\nHosted: candidate image -> jobs/migrate_markets.py -> API rollout")
+    print(f"The Job calls metatables.upgrade_application({PROVIDER!r}) on every deployment.")
+    print("MetaTables >=0.1.31 batches inspections and waits up to 300s for a waking API.")
+    print("At head, catalog reconciliation still runs; migrated=False is not a skip signal.")
+    print("INFO logs from metatables.migrations.runner report each phase's elapsed time.")
+    print("Consumers migrate only their own providers, not the ms-markets provider.")
+
+
+if __name__ == "__main__":
+    main()

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ms-markets"
-version = "2.2.6.dev26"
+version = "2.2.6.dev27"
 description = "Financial markets ORM, engines, and application management extensions for Main Sequence."
 readme = "README.md"
 requires-python = ">=3.13,<3.14"
@@ -37,7 +37,7 @@ dependencies = [
   "cachetools",
   "fastapi>=0.136.3",
   "mainsequence>=9.0.19,<10",
-  "mainsequence-metatable>=0.1.29,<0.2",
+  "mainsequence-metatable>=0.1.31,<0.2",
   "numpy",
   "pandas",
   "pip>=26.1.1",

ms_markets-2.2.6.dev27/tests/msm/maintenance/test_migration_job.py

@@ -0,0 +1,57 @@
+from __future__ import annotations
+
+import logging
+import runpy
+from pathlib import Path
+
+import metatables
+import pytest
+
+
+PROJECT_ROOT = Path(__file__).resolve().parents[3]
+JOB_PATH = PROJECT_ROOT / "jobs" / "migrate_markets.py"
+
+
+@pytest.mark.parametrize("migrated", [False, True])
+def test_job_always_calls_shared_upgrade_and_exposes_timings(
+    monkeypatch, capsys, caplog, migrated: bool
+) -> None:
+    calls = []
+
+    def upgrade(provider):
+        calls.append(provider)
+        logging.getLogger("metatables.migrations.runner").info(
+            "Application migration msm:mainsequence.markets: catalog finalization took 0.001s"
+        )
+        return {"revision": "0018", "migrated": migrated, "data_source_uid": "test-source"}
+
+    monkeypatch.setattr(metatables, "upgrade_application", upgrade)
+    # Restore the logger's level after running the script in this process.
+    runner_logger = logging.getLogger("metatables.migrations.runner")
+    monkeypatch.setattr(runner_logger, "level", logging.WARNING)
+
+    runpy.run_path(str(JOB_PATH), run_name="__main__")
+
+    assert calls == ["msm_migrations:migration"]
+    assert "catalog finalization took 0.001s" in caplog.text
+    output = capsys.readouterr().out
+    assert "Applying and reconciling msm_migrations:migration" in output
+    state = "migrated" if migrated else "already current"
+    assert f"0018 ({state}) on test-source" in output
+
+
+def test_job_does_not_hide_migration_or_finalization_failure(monkeypatch, capsys) -> None:
+    failure = RuntimeError("catalog finalization failed")
+
+    def upgrade(provider):
+        assert provider == "msm_migrations:migration"
+        raise failure
+
+    monkeypatch.setattr(metatables, "upgrade_application", upgrade)
+    monkeypatch.setattr(logging.getLogger("metatables.migrations.runner"), "level", logging.WARNING)
+
+    with pytest.raises(RuntimeError) as raised:
+        runpy.run_path(str(JOB_PATH), run_name="__main__")
+
+    assert raised.value is failure
+    assert "already current" not in capsys.readouterr().out

{ms_markets-2.2.6.dev26 → ms_markets-2.2.6.dev27}/tests/msm/test_package_metadata.py

@@ -39,8 +39,8 @@ def test_package_metadata_enforces_sdk_9_floor_without_exact_patch_pin() -> None
 def test_package_metadata_requires_extracted_metatables_client() -> None:
     requirement = _requirement("mainsequence-metatable")
 
-    assert Version("0.1.28") not in requirement.specifier
-    assert Version("0.1.29") in requirement.specifier
+    assert Version("0.1.30") not in requirement.specifier
+    assert Version("0.1.31") in requirement.specifier
     assert Version("0.1.99") in requirement.specifier
     assert Version("0.2.0") not in requirement.specifier
     assert all(specifier.operator != "==" for specifier in requirement.specifier)

ms_markets-2.2.6.dev26/examples/msm/platform/metatable_migration_lifecycle.py

@@ -1,21 +0,0 @@
-from __future__ import annotations
-
-PROVIDER = "msm_migrations:migration"
-
-
-def main() -> None:
-    print("ms-markets MetaTable migrations use the application-owned MetaTables provider:")
-    print(PROVIDER)
-
-    print("\nAdmin command sequence:")
-    for command in (
-        f"metatables --json migrations current --provider {PROVIDER}",
-        f'metatables migrations revision --provider {PROVIDER} -m "describe change"',
-        f"metatables --local migrations upgrade --provider {PROVIDER} head",
-        f"metatables --local migrations downgrade --provider {PROVIDER} <revision>",
-    ):
-        print(command)
-
-
-if __name__ == "__main__":
-    main()