ms-markets 2.2.6.dev30__tar.gz → 2.2.7.dev31__tar.gz

{ms_markets-2.2.6.dev30 → ms_markets-2.2.7.dev31}/CHANGELOG.md

@@ -7,6 +7,22 @@ and this project follows versioned releases.
 
 ## [Unreleased]
 
+## [2.2.7] - 2026-10-11
+
+### Fixed
+
+- Restore a forward upgrade for populated PostgreSQL databases at the retired
+  `0017` head, which blocked the Production API rollout. A compatibility anchor
+  and merge revision `0020` join that history with development's `0018`/`0019`
+  without changing those applied revisions or replacing application tables.
+- Validate all 62 application table column contracts before reconciliation;
+  refuse overlong identifiers, bound only ten known legacy string columns, and
+  verify locked tables' row counts and checksums remain unchanged. Extra tables
+  and columns are preserved. Already-bounded development needs no alterations.
+- Add original-history PostgreSQL data-preservation and failure regressions,
+  fresh/development upgrade checks, and matching migration guide, release
+  instructions, ADR 0046, tutorial, and print-only lifecycle example.
+
 ## [2.2.6] - 2026-10-10
 
 ### Added

{ms_markets-2.2.6.dev30 → ms_markets-2.2.7.dev31}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ms-markets
-Version: 2.2.6.dev30
+Version: 2.2.7.dev31
 Summary: Financial markets ORM, engines, and application management extensions for Main Sequence.
 Project-URL: Changelog, https://github.com/mainsequence-projects/MainSequenceMarkets/blob/main/CHANGELOG.md
 Project-URL: Source, https://github.com/mainsequence-projects/MainSequenceMarkets

{ms_markets-2.2.6.dev30 → ms_markets-2.2.7.dev31}/docs/ADR/0045-squashed-initial-schema-and-namespaced-packages.md

@@ -14,6 +14,10 @@ Amends ADRs [0034](0034-command-center-asset-monitor-helpers.md),
 [0043](0043-sdk-9-metatables-client-hard-cut.md), and withdraws the 2.0.2
 CHANGELOG promise that revisions `0001`–`0017` stay unchanged.
 
+The legacy upgrade policy below is superseded by
+[ADR 0046](0046-legacy-0017-schema-bridge.md) in 2.2.7: Production was found
+with a populated `0017` schema, which must be preserved.
+
 ## Context
 
 No MetaTables runtime holds an applied ms-markets history; `0001`–`0017` only
@@ -39,11 +43,10 @@ MetaTables local SQLite runtime (#15). The wheel also still installed top-level
 
 ## Consequences
 
-- A database at any of `0001`–`0017` fails with Alembic's unknown-revision error.
-  For a local runtime, with 2.0.x still installed, run
-  `metatables --local migrations downgrade base --provider msm_migrations:migration`,
-  then install 2.1.0 and upgrade from empty. Hosted environments are migrated
-  only by the deployment workflow's migration Job.
+- The original release rejected `0001`–`0017`. Production later proved the
+  assumption above false: its populated database remained at `0017`.
+  Release 2.2.7 provides the forward bridge in ADR 0046. Do not reset that
+  database or stamp its version. Earlier legacy heads require separate analysis.
 - Imports of `cli` and `command_center` change to `msm_cli` and
   `msm_command_center`.
 - Later schema changes are new revisions after `0018`. MetaTables generates them

ms_markets-2.2.7.dev31/docs/ADR/0046-legacy-0017-schema-bridge.md

@@ -0,0 +1,68 @@
+# 0046. Preserve The Populated Legacy 0017 Schema
+
+## Status
+
+Accepted for 2.2.7. Amends the legacy upgrade policy in
+[ADR 0045](0045-squashed-initial-schema-and-namespaced-packages.md).
+
+## Context
+
+Production's migration Job fails with `Can't locate revision identified by
+'0017'`. Its tables contain production data; resetting them or stamping a newer
+revision would respectively destroy data or skip schema verification.
+Development already uses the `0018`/`0019` graph and must keep working.
+
+Replaying the original revisions from commit
+`29dc96b693baa321dd8a0254c885407428378597` into disposable PostgreSQL produces
+62 application tables. They match the frozen `0018` column baseline except for
+ten unbounded identifier strings in OrdersTS, TradesTS, PortfolioWeightsTS and
+SignalWeightsTS. Production's physical column inventory confirms these differences.
+
+## Decision
+
+Keep applied `0018` and `0019` byte for byte. Add a no-op `0017` compatibility
+anchor whose parent is `0018`, then merge `0017` and `0019` at `0020`:
+
+```text
+base -> 0018 -> 0019 ---------------------> 0020
+          |                                  ^
+          +----> 0017 compatibility anchor ---+
+```
+
+The anchor maps the legacy schema to its verified baseline; it does not replay
+the original `0017` DDL. A database already at `0017` skips initial table creation
+and receives `0019` plus `0020`. Fresh databases and development histories also
+reach one head, using the unchanged standard MetaTables migration environment.
+Only the legacy head `0017` is recognized, not earlier legacy revisions.
+
+`0020` checks every expected table, column, type and nullability before changing
+anything. Extra tables and columns are preserved. It accepts only the ten known
+unbounded strings as differences. PostgreSQL locks the affected tables, refuses
+any value exceeding 255 characters, alters the limits in place, and compares row
+counts and two 64-bit sums of row MD5 hashes before and after. A mismatch fails
+the transaction. Logs report counts and checksum equality without row values.
+SQLite checks the baseline without physical string-limit alterations.
+Already-bounded development tables require no alterations or locks.
+
+Hosted execution remains the candidate image's migration Job, followed by API
+rollout only after successful catalog finalization. No local session runs hosted
+DDL. Both new revisions reject downgrade; future corrections use new revisions.
+
+## Validation
+
+The PostgreSQL regression fixture is a schema-only dump of the original `0017`
+history. Tests seed all 62 application tables, prove overlong values fail without
+changing rows or the revision, and compare every table's row multiset before and
+after upgrading twice. Fresh, `0018`, and `0019` paths are also exercised, and a
+missing required column fails validation. The frozen baseline ships in the wheel.
+Release verification must include development's migration and API health before
+promoting to Production, then Production's migration and both sites' API access.
+
+## Consequences
+
+The graph explicitly carries a compatibility mapping for an already-squashed
+history. It cannot validate a falsely stamped or otherwise drifted database by
+its revision alone; the frozen column guards provide that check. Identifier
+values over 255 characters require a separate forward repair, never truncation.
+The ledger uniqueness correction in `0019` remains the only constraint removal;
+no application tables, columns, or rows are removed by this bridge.

{ms_markets-2.2.6.dev30 → ms_markets-2.2.7.dev31}/docs/ADR/README.md

@@ -55,3 +55,5 @@ Recommended sections:
 - [0002. Command Center Adapter Discovery](fast_api/v1/0002-command-center-adapter-discovery.md)
 - [0003. Fixed Income Pricer API](fast_api/v1/0003-fixed-income-pricer-api.md)
 - [0004. Reusable Delete Impact Contract](fast_api/v1/0004-delete-impact-contract.md)
+
+- [0046. Preserve The Populated Legacy 0017 Schema](0046-legacy-0017-schema-bridge.md)

{ms_markets-2.2.6.dev30 → ms_markets-2.2.7.dev31}/docs/knowledge/msm/migrations/index.md

@@ -22,11 +22,21 @@ does not change the provider key `msm:<namespace>` or the
 `ms_markets__alembic_version` table. See
 [ADR 0044](../../../ADR/0044-namespaced-migration-provider-package.md).
 
-Since 2.1.0 the history starts at `0018_initial_schema`, which replaces
-`0001`–`0017` without reusing their IDs. A database left at any of those
-revisions fails with an unknown-revision error: downgrade it to `base` with
-2.0.x installed, then apply 2.1.0 from empty. See
-[ADR 0045](../../../ADR/0045-squashed-initial-schema-and-namespaced-packages.md).
+Since 2.1.0 fresh databases start at `0018_initial_schema`. Release 2.2.7
+restores recognition of the retired `0017` head through a compatibility anchor
+and merge revision `0020`. A populated PostgreSQL database at `0017` upgrades
+in place; databases at `0018` or `0019` reach the same head. Applied revisions
+`0018` and `0019`, provider identity, and version-table binding are unchanged.
+
+The bridge validates all 62 application tables and their column contracts.
+Only ten known, formerly unbounded identifier columns may differ. It refuses
+values longer than 255 characters before bounding those columns, locks affected
+tables against writes, and verifies row counts and row multiset checksums before
+and after the change. Extra tables and columns remain intact. Already-current
+development columns need no alteration. No reset, downgrade, or manual stamp is
+needed. Revisions `0001`–`0016` remain unsupported; investigate those separately.
+The bridge is forward-only. See
+[ADR 0046](../../../ADR/0046-legacy-0017-schema-bridge.md).
 
 Revision `0019_ledger_hypertable_uniqueness` removes the event ledger's extra
 unique constraint that omitted the TimescaleDB partition key. The full-grain

{ms_markets-2.2.6.dev30 → ms_markets-2.2.7.dev31}/docs/releasing.md

@@ -129,6 +129,30 @@ rebuilds the image, runs the `migrate-markets` Job and redeploys the API. Pushes
 `main` happen only through release merges, so the `main` deployment follows
 stable releases.
 
+### Legacy 0017 compatibility
+
+Release 2.2.7 bridges populated PostgreSQL databases at `0017` to `0020` in
+place. `0018` and `0019` remain unchanged, and development at either revision
+also reaches `0020`. Never reset, downgrade, or stamp a production database to
+recover from the retired-history error. See
+[the migration guide](knowledge/msm/migrations/index.md) and
+[ADR 0046](ADR/0046-legacy-0017-schema-bridge.md).
+
+In addition to the service-free checks, run the original-history regression
+against an empty disposable PostgreSQL database on localhost:
+
+```bash
+MS_MARKETS_LEGACY_TEST_POSTGRES_URL=postgresql+psycopg2://localhost/markets_migration_test \
+  uv run --frozen --all-extras pytest -q tests/msm/maintenance/test_legacy_schema_migration.py
+```
+
+Confirm the development migration succeeds and its API remains healthy before
+merging into `main`. After the Production workflow, verify the Job's preserved
+row counts/checksums and successful catalog finalization, the active API revision,
+and delegated requests from both static sites to their own environment APIs.
+A failed guard blocks rollout; investigate the drift or overlong values instead
+of bypassing the check. The bridge's new revisions are forward-only.
+
 ### Upgrading the migration client
 
 The project requires `mainsequence-metatable>=0.1.33,<0.2`; `uv.lock` and

{ms_markets-2.2.6.dev30 → ms_markets-2.2.7.dev31}/docs/tutorial/index.md

@@ -26,6 +26,11 @@ helpers for time-indexed facts).
    been finalized. A consuming project uses the schema owned by the ms-markets
    deployment and migrates only its own providers.
 
+A populated PostgreSQL schema at the legacy `0017` head upgrades in place to
+`0020` with 2.2.7 or later. Development at `0018` or `0019` reaches the same head.
+Do not reset or stamp the old database; the bridge validates its columns and
+preserves rows. Overlong identifiers fail safely for a separate forward repair.
+
 Run `python examples/msm/platform/metatable_migration_lifecycle.py` for a safe,
 print-only walkthrough. The shared runner batches inspections automatically and
 reconciles the catalog even at the current revision. Changed, non-empty authored

{ms_markets-2.2.6.dev30 → ms_markets-2.2.7.dev31}/examples/msm/platform/metatable_migration_lifecycle.py

@@ -12,10 +12,16 @@ def main() -> None:
         f"metatables --local --json migrations current --provider {PROVIDER}",
         f'metatables --local migrations revision --provider {PROVIDER} -m "describe change"',
         f"metatables --local migrations upgrade --provider {PROVIDER} head",
-        f"metatables --local migrations downgrade --provider {PROVIDER} <revision>",
     ):
         print(command)
 
+    print("\nFresh, legacy 0017, development 0018 and 0019 all upgrade to head 0020.")
+    print("0018/0019 stay unchanged; 0020 checks all 62 table column contracts.")
+    print("Legacy PostgreSQL limits change only if all identifiers fit 255 characters.")
+    print("Affected tables are locked; row counts and checksums must remain unchanged.")
+    print("Extra tables/columns remain intact. No reset, stamp, or downgrade is needed.")
+    print("The compatibility anchor and merge are forward-only; use new repairs.")
+
     print("\nHosted: candidate image -> jobs/migrate_markets.py -> API rollout")
     print(f"The Job calls metatables.upgrade_application({PROVIDER!r}) on every deployment.")
     print("MetaTables >=0.1.33 batches inspections and waits up to 300s for a waking API.")

{ms_markets-2.2.6.dev30 → ms_markets-2.2.7.dev31}/mkdocs.yml

@@ -123,6 +123,7 @@ nav:
       - SDK 9 And MetaTables Client Hard Cut: ADR/0043-sdk-9-metatables-client-hard-cut.md
       - Namespaced Migration Provider Package: ADR/0044-namespaced-migration-provider-package.md
       - Squashed Initial Schema And Namespaced Packages: ADR/0045-squashed-initial-schema-and-namespaced-packages.md
+      - Preserve The Populated Legacy 0017 Schema: ADR/0046-legacy-0017-schema-bridge.md
       - Command Center Asset Monitor Helpers: ADR/0034-command-center-asset-monitor-helpers.md
       - FastAPI v1:
           - Calendar CRUD Route: ADR/fast_api/v1/0001-calendar-crud-route.md

{ms_markets-2.2.6.dev30 → ms_markets-2.2.7.dev31}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ms-markets"
-version = "2.2.6.dev30"
+version = "2.2.7.dev31"
 description = "Financial markets ORM, engines, and application management extensions for Main Sequence."
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

ms_markets-2.2.7.dev31/src/msm_migrations/versions/mainsequence_markets/0017_legacy_schema_anchor.py

@@ -0,0 +1,23 @@
+"""Recognize the retired 0017 head as the schema equivalent of the 0018 baseline.
+
+This is an anchor, not a replay of the retired 0001-0017 DDL. The original
+0017 tables and 0018 tables have identical columns except ten unbounded
+identifiers. Revision 0020 checks the complete column baseline, bounds those
+identifiers only after checking every value, and merges both histories.
+
+0018 and 0019 stay byte-for-byte unchanged. A fresh database still runs 0018;
+one already at 0017 skips that table-creation revision. Both must reach 0020.
+"""
+
+revision: str = "0017"
+down_revision: str = "0018"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    """The physical schema already exists; 0020 performs the compatibility check."""
+
+
+def downgrade() -> None:
+    raise RuntimeError("The legacy schema anchor is forward-only; use a new migration.")