mainsequence-metatable 0.1.8.dev17__tar.gz → 0.1.9__tar.gz

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.9}/.agents/skills/client/metatables-local-development/SKILL.md

@@ -72,6 +72,8 @@ Do not retry against automatic hosted discovery when a local connection fails.
    Substitute the application's own module. Provider code and revisions are
    installed in the application process; the API requires no provider approval
    or installation. System initialization and application histories remain separate.
+   Running the provider yourself is for the local runtime only; hosted runtimes
+   receive the same revisions from the deployment workflow's migration Job.
 5. Build application contracts/queries with the
    [table skill](../metatables-meta-tables/SKILL.md), or producers/readers with the
    [updater skill](../metatables-time-index-table-updates/SKILL.md). Use small,
@@ -134,26 +136,36 @@ integration-test fixture.
 ## Finish verification and switch back
 
 Record what passed, the runtime/DataSource used, and any engine-specific work still
-unverified. Carry forward the reviewed application code and migration revisions;
+unverified. Commit the reviewed application code and its migration revisions together;
 local rows, catalog UIDs, credentials and fixture data are not promoted by a switch.
+The application's deployment workflow applies those revisions to the hosted runtime
+before the code rolls out; see the [migrations skill](../metatables-migrations/SKILL.md)
+and `docs/client/deploy-application-migrations.md`. Add the migration Job if the
+workflow lacks one.
 
 When returning to the environment is part of the requested workflow:
 
 1. Finish local tests and stop active writers. In Admin **Settings → Runtime mode**,
-   inspect the displayed hosted environment and choose the intended registered
-   DataSource. Use **Switch to Hosted**. This changes the API worker and selected
-   database; it does not copy or merge the local database. Leave local capability
-   enabled if the developer needs to switch back later.
+   inspect the displayed hosted environment and use **Switch to Hosted**. Hosted
+   opens the runtime database the API's deployment declares, through the same
+   Environment Secret; there is no DataSource to choose and the launcher never
+   migrates it. This changes the API worker and database; it does not copy or merge
+   the local database. Leave local capability enabled if the developer needs to
+   switch back later.
 2. Re-read runtime status and verify `local_mode: false`, the intended verified
-   hosted environment, the expected DataSource and readiness. The same local
+   hosted environment, the expected DataSource and readiness. `migration_required`
+   means the branch has system migrations the deployed API does not have yet; it
+   clears once that code is deployed. The same local
    connection command still reaches this supervised API; its `--local` flag alone
    does not prove SQLite. If the switch fails and the old worker is restored,
    report the actual mode instead of claiming the switch succeeded.
 3. Restart application/test client processes and resolve catalog bindings again.
    Do not reuse local table UIDs or already-bound updater instances against the
-   environment database. Apply reviewed revisions or perform environment writes
-   only within the user's requested scope; passing local tests is not itself a
-   request to seed fixtures, run backfills or migrate that database.
+   environment database. Do not apply application revisions from this session;
+   the deployment's migration Job does. `migrations current` can confirm the
+   deployed revision. Perform other environment writes only within the user's
+   requested scope; passing local tests is not itself a request to seed fixtures
+   or run backfills.
 
 If the user instead wants to connect to a separately deployed API, follow the
 installation guide's hosted endpoint selection. Restore prior connection settings

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.9}/.agents/skills/client/metatables-meta-tables/SKILL.md

@@ -35,8 +35,9 @@ identity and Git source facts use SDK imports. Table operations use the client's
 with SQLite behind it; ordinary clients never open local table files. Local
 workspaces use one persistent SQLite database for system and user tables per
 checkout; Git branches share it.
-Settings explicitly initializes or selects the runtime DataSource in both modes;
-startup never runs Alembic. See `docs/operations/catalog-migrations.md`.
+A hosted API's deployment declares and initializes its runtime DataSource; Local
+Settings explicitly initializes or selects the SQLite file. Startup never runs
+Alembic. See `docs/operations/catalog-migrations.md`.
 
 ## Authoring decisions
 
@@ -118,6 +119,12 @@ A null cutoff requires a scope. Do not replace that workflow with ad hoc SQL.
 Deletion/cascade requires edit access, relevant protection checks, and explicit
 cascade intent; it is not an administrator-role bypass or migration repair.
 
+On a TimescaleDB DataSource, time-index tables are hypertables. Their compression
+and retention policies are set only through `TimeIndexMetaTable.set_timescale_policies`
+(Writer) and read with `get_timescale_policies`. Do not add policies to table
+contracts or Alembic revisions; to keep them in code, call the client from a
+bootstrap Job. Retention permanently drops older chunks, including backfills.
+
 ## Validation
 
 Check authored models, normalization, physical naming, FK targets, and time grain

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.9}/.agents/skills/client/metatables-migrations/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: metatables-migrations
-description: "Create and evolve managed application tables through Alembic migration providers with the installed MetaTables Python client. Covers provider scope and placement, scaffolding, revision authoring, client-side execution, and reservation or finalization failures. Excludes API catalog migrations, client-library implementation, API/server changes and repository tooling."
+description: "Create and evolve managed application tables through Alembic migration providers with the installed MetaTables Python client. Covers provider scope and placement, scaffolding, revision authoring, local execution, the deployment-workflow migration Job for hosted runtimes, and reservation or finalization failures. Excludes API catalog migrations, client-library implementation, API/server changes and repository tooling."
 ---
 
 # MetaTables client application migrations
@@ -34,9 +34,38 @@ finalize catalog bindings through the API. Use the
 [table skill](../metatables-meta-tables/SKILL.md) for contract design.
 
 Check `docs/reference/capabilities.md` and `metatables migrations --help` before
-promising a command. The API's own catalog migrations are a separate history that
-an admin runs from Settings; see `docs/operations/catalog-migrations.md`. They
-are not application-provider revisions.
+promising a command. The API's own catalog migrations are a separate history: a
+hosted API's deployment applies them before it rolls out, and Local mode runs them
+from Settings; see `docs/operations/catalog-migrations.md`. They are not
+application-provider revisions.
+
+## Recommended workflow
+
+Follow this sequence, which mirrors how the MetaTables API deploys its own system
+migrations. Read `docs/client/deploy-application-migrations.md`.
+
+1. **Develop locally.** Author the revision and apply it to verified local SQLite
+   with the [local development skill](../metatables-local-development/SKILL.md).
+2. **Commit the revision with the code that needs it.**
+3. **Deploy.** The application's `.mainsequence/workflows/` file declares a
+   migration Job that calls `upgrade_application` for each provider, in
+   dependency order, from the candidate image. Every deployed resource that reads
+   or writes the tables (FastAPI, `harness_agent` and others) has a deploy step
+   with `needs: [migrate]`. A failed migration blocks the rollout.
+
+Rules:
+
+- Hosted runtimes are migrated only by that Job. Do not run `migrations upgrade`
+  or `downgrade` against a hosted API from a developer or agent session. If the
+  user explicitly requests it, do it once and recommend adding the Job.
+- When the application has no migration Job yet, add one alongside its first
+  managed table or its MetaTables upgrade.
+- Do not migrate at application startup: several pods would race, and a failure
+  would not stop the rollout.
+- Keep each revision usable by the release currently deployed, which keeps serving
+  during and after a failed rollout. Add first; drop or rename in a later release.
+- Redeploying an older image does not roll back the schema. Downgrade is never a
+  deployment step.
 
 ## Source and runtime context
 
@@ -112,7 +141,8 @@ the wrong revisions.
 
 ## Execute with the environment connection
 
-After verifying the intended runtime, use the application's provider reference:
+During development, after verifying the local runtime, use the application's
+provider reference:
 
 ```bash
 metatables --local migrations upgrade --provider ledger.migrations:migration
@@ -131,9 +161,9 @@ API Writer checks govern connection admission and provider catalog operations.
 The client reserves catalog entries, runs Alembic, closes its physical connection,
 and finalizes contracts through the API. Treat connection material as private.
 
-Use the application's setup entry point when available. Python code can call
-`metatables.upgrade_application("ledger.migrations:migration")`;
-`src/metatables/examples/scripts/setup_metatables.py` shows the pattern. Remove the retired
+Python code can call `metatables.upgrade_application("ledger.migrations:migration")`;
+`src/metatables/examples/scripts/setup_metatables.py` shows the pattern, and the
+deployment migration Job uses the same call. Remove the retired
 `application_migration_providers` setting and use Python references instead of aliases.
 Application migration histories remain separate from API system migrations.
 
@@ -180,5 +210,6 @@ Read `docs/operations/recovery-and-observability.md`.
 ## Validation
 
 Verify provider scope and placement, revision content, the selected environment
-connection, repeat execution, and final active bindings. Report separately what was checked
-offline and what was exercised against a configured API.
+connection, repeat execution, and final active bindings. Check that the deployment
+workflow runs the migration Job before every resource that uses the tables. Report
+separately what was checked offline and what was exercised against a configured API.

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.9}/.agents/skills/client/metatables-upgrade-legacy-app/SKILL.md

@@ -96,6 +96,9 @@ Use the [application migrations skill](../metatables-migrations/SKILL.md),
 `docs/client/define-and-migrate-tables.md`, and
 `docs/adr/api/0013-application-owned-migrations.md` for that workflow. Ordinary
 SQL continues through the governed API, independently of the migration connection.
+If the application's deployment workflow does not yet run its providers in a
+migration Job before its API and agents roll out, add one as
+`docs/client/deploy-application-migrations.md` describes.
 
 ## Verify the upgrade
 

{mainsequence_metatable-0.1.8.dev17/src/mainsequence_metatable.egg-info → mainsequence_metatable-0.1.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.8.dev17
+Version: 0.1.9
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.9}/docs/adr/api/0001-unified-api-storage-and-local-sqlite.md

@@ -30,6 +30,8 @@ Amended 2026-09-29: one enforced runtime binding and explicit bootstrap select o
 database for system and user tables.
 Amended 2026-10-02: each runtime mode lists and selects only its own DataSources,
 and inspection checks the login's prerequisites before setup.
+Amended 2026-10-03: the deployment declares the hosted runtime database; Settings
+is read-only in Hosted mode.
 
 Companion decision: [Main Sequence SDK ADR 0035: Independent Git source and platform execution context](https://github.com/mainsequence-sdk/mainsequence-sdk/blob/metatables_removal/docs/adr/0035-independent-git-source-and-platform-context.md).
 The SDK prerequisite is implemented on `metatables_removal`; the current local
@@ -83,8 +85,67 @@ API requirement. The guides and examples describe the implemented shared path.
 
 ## Decision
 
+### The deployment declares the hosted runtime database (amended 2026-10-03)
+
+Implemented 2026-10-03 for [MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13) in the API, the migration Job and the Admin.
+The PostgreSQL transition test and the container suite were updated but not run;
+hosted verification is pending.
+
+Settings used to save the hosted selection in the database it pointed to, with a
+pointer file on the pod. A restarted or additional pod lost that pointer. The
+hosted runtime database is now declared by the deployment:
+
+- **Declaration.** The API's `configuration.yaml`
+  (`metatables/api/metatables/configuration.yaml`) has a `runtime_database`
+  section with the engine, the name of the Environment Secret holding the
+  connection URI, the default schema, the TLS mode and optional certificate
+  Secret names. It holds names and settings, never secret values. Each branch
+  keeps its own file, so each Environment's deployment reads its own file and
+  Secret.
+- **Secret.** The database administrator creates it in each Environment, for
+  example `postgresql://login:password@host:5432/database`. Host, port, database,
+  login and password come only from the URI, certificates only from the named
+  Secrets. The runtime DataSource's `password_secret_uid` is the URI Secret's UID
+  ([ADR 0009](0009-shared-credential-store.md)).
+- **Deployment.** On every deployment the `migrate-system` Job reads the
+  declaration and the Secret. It checks the prerequisites, applies the system
+  migrations, registers the runtime DataSource and reapplies access setup
+  ([ADR 0014](0014-main-sequence-release-jobs-and-production-migrations.md)).
+  Pointing the Secret at another database follows the same procedure: that
+  database is initialized and becomes the runtime. Each database keeps its own
+  catalog, and nothing is transferred. When only the connection changes (host,
+  login, TLS), the deployment updates the registration.
+- **API.** Every pod reads the same declaration and Secret at startup and takes
+  the runtime row from the catalog. It never runs DDL and writes no pointer, so
+  restarts and additional pods agree and report the same instance identity
+  ([MetaTables #14](https://github.com/mainsequence-projects/MetaTables/issues/14)).
+  A pod that starts on an unprepared database reports `migration_required` or
+  `registration_required` and serves no application operations. A Secret change
+  therefore takes effect on deployment.
+- **Settings, Hosted mode.** Settings is read-only: it shows the declaration, the
+  resolved connection, the status and the revisions.
+  - Hosted `configure`, `migrate` and `activate` answer 409.
+  - `select` and the pre-activation Hosted registry
+    (`runtime-source-candidates.json`, `hosted_source_uid`) are removed.
+  - Hosted DataSources are registered only in the active runtime catalog.
+  - Settings does not show the migration Job's result; the deployment run
+    reports a failed Job.
+- **Developer launcher, Hosted mode.** The launcher reads the packaged deployment
+  file and the same Secret, so it uses the same database; its own
+  `configuration.yaml` only grants the developer capability. It never migrates. A
+  newer local migration reports `migration_required` until it is deployed.
+
+Local mode is unchanged. For Hosted mode, this supersedes three earlier
+decisions:
+- the Hosted registry and selection in the 2026-10-02 amendment;
+- Settings-initiated hosted initialization in the 2026-09-29 bootstrap amendment;
+- the persisted hosted selection in the 2026-09-29 bootstrap amendment.
+
 ### Each runtime mode selects only its own DataSources (amended 2026-10-02)
 
+Hosted rows are superseded by the 2026-10-03 amendment above: Hosted has no
+registry before activation and no selection.
+
 Implemented 2026-10-02 in the API and Admin; the PostgreSQL runtime-transition test
 was updated but not run.
 
@@ -160,6 +221,9 @@ binding, automatic local catalog upgrades, and the requirement that switching to
 Hosted already have a migrated catalog. It corrects this ADR's runtime lifecycle;
 it does not introduce a second execution architecture.
 
+For Hosted mode, the 2026-10-03 amendment replaces Settings selection and
+initialization with the deployment's declaration and migration Job.
+
 Settings selects the runtime mode and its DataSource. That database contains both
 MetaTables' system tables and user tables. Local selects one persistent SQLite
 file; Hosted selects PostgreSQL/TimescaleDB. MySQL and MSSQL remain connection-only
@@ -274,10 +338,18 @@ The supervisor passes mode and transport parameters explicitly to the child;
 `METATABLES_LOCAL_RUNTIME` has no authority. Database credentials remain private
 deployment inputs and are never stored in YAML or sent to Vite.
 
-The runtime descriptor reports capability, switching availability and a fresh
-instance identity. Admin requests carry that identity so stale requests are
-rejected after a restart. During transition the UI blocks work, reloads runtime
-context and remounts cached views. A mode change affects the entire developer API
+The runtime descriptor reports capability, switching availability and an
+instance identity. Admin requests carry that identity so requests made against a
+previous runtime are rejected. During transition the UI blocks work, reloads runtime
+context and remounts cached views.
+
+Amended 2026-10-03 for [MetaTables #14](https://github.com/mainsequence-projects/MetaTables/issues/14):
+the instance identity names the runtime binding, not the process. It is derived
+from the mode, the runtime database's public configuration, the runtime
+DataSource UID, its activation and the catalog revision. Every process and pod
+serving one binding reports the same identity, so an API deploy or restart keeps
+open pages valid; selecting another DataSource, activating it, applying
+migrations or switching mode changes it. A mode change affects the entire developer API
 instance, never an individual user's view of a shared service.
 
 ### Enforced runtime binding (amended 2026-09-29)

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.9}/docs/adr/api/0009-shared-credential-store.md

@@ -9,6 +9,10 @@
 
 Date: 2026-09-30
 
+Amended 2026-10-03 ([MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13)): the hosted runtime login's password comes from the
+URI Secret that the deployment declares. See
+[Bootstrap, restart and lifecycle](#bootstrap-restart-and-lifecycle).
+
 Status: Accepted. Implemented by the API CredentialStore providers and catalog
 revision `0006_local_credentials`.
 
@@ -131,17 +135,21 @@ coverage must additionally run on its corresponding host.
 ### Bootstrap, restart and lifecycle
 
 Local SQLite opens using the existing API-owned file binding, before credential
-lookup. Hosted bootstrap resolves the selected connection's UUID through SDK
-Secrets before opening the catalog. Its credential never depends on opening the
-database that needs that password. Saved selection JSON holds only public
-settings and UUID references.
+lookup. Hosted bootstrap reads the Environment Secret named by
+`runtime_database.uri_secret` (amended 2026-10-03). It reads the Secret by name
+through the SDK before opening the catalog. The login and password come from that
+URI, and the runtime DataSource references the Secret's UID as
+`password_secret_uid`. The hosted store serves that UID only as the database
+password and refuses to delete it. Every other reference still resolves through
+SDK Secrets by UID. The credential never depends on opening the database that
+needs it. Local selection JSON holds only public settings and UUID references.
 
 A fresh local catalog must receive explicit system migrations before saving
 sources/credentials. Local saves never substitute JSON for encrypted SQL storage.
 Previously saved local candidate metadata can be materialized by explicit setup,
-retaining DataSource UIDs. Hosted pre-catalog registration continues to use SDK
-Secrets and its existing metadata registry; it becomes the runtime row during
-explicit setup. Startup never silently applies migrations.
+retaining DataSource UIDs. Hosted has no pre-catalog registration: the
+deployment's migration Job registers the declared runtime, and other hosted
+sources are registered in its catalog. Startup never silently applies migrations.
 
 `metatables credentials` provides admin-only local operations:
 

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.9}/docs/adr/api/0014-main-sequence-release-jobs-and-production-migrations.md

@@ -3,6 +3,8 @@
 Date: 2026-10-02
 
 Status: Accepted. Implemented on `development`; hosted verification pending.
+Amended 2026-10-03 ([MetaTables #13](https://github.com/mainsequence-projects/MetaTables/issues/13)): the Job reads the declared runtime database itself
+and initializes it on the first deployment.
 
 Owners: MetaTables API. Main Sequence owns Job execution, images, deployment and
 platform authentication.
@@ -20,6 +22,31 @@ its API serves.
 
 ## Decision
 
+### The Job prepares the declared database (amended 2026-10-03)
+
+The Job no longer reads the running API. `metatables runtime upgrade` loads
+`runtime_database` from the candidate image's `configuration.yaml` and the
+Environment Secret it names ([ADR 0001](0001-unified-api-storage-and-local-sqlite.md)).
+Every run executes the same steps:
+
+1. Check that the login can secure the database (ADR 0007 prerequisites).
+2. Apply the system migrations.
+3. Register the runtime DataSource, or update its connection to the declaration.
+4. Reapply access setup. Per-User role passwords derive from the login password,
+   so a rotated password reaches them on the next deployment.
+
+The first deployment therefore initializes the database. A missing declaration
+or Secret, an unreachable database or unmet prerequisites fail the Job and block
+the rollout. The deployment workflow is the authority, so no admin caller is
+required. A new runtime DataSource records the Job's SDK user as its creator.
+The Job prints `initialized`, `upgraded` or `up_to_date` with the revisions.
+
+Settings' **Run MetaTables migrations** and `metatables runtime initialize` remain
+for Local mode only. The original decision follows. The amendment supersedes
+steps 1–2, the skip rules and the Settings initialization paragraph.
+
+### Original decision (2026-10-02)
+
 The deployment workflow upgrades the runtime from the candidate image before the
 rollout:
 

mainsequence_metatable-0.1.9/docs/adr/api/0015-timescale-compression-and-retention-policies.md

@@ -0,0 +1,192 @@
+# ADR 0015: TimescaleDB compression and retention policies
+
+Date: 2026-10-02
+
+Status: Accepted.
+
+Implementation status: Implemented on the `timescale-policies` branches of the API
+and the Admin, with focused tests. TimescaleDB container verification is pending.
+
+Owner: MetaTables API. The Admin owns the screens described here.
+
+Related decisions: [ADR 0002: Table ownership](0002-application-administration-and-table-ownership.md),
+[ADR 0007: Database-enforced access](0007-database-enforced-table-access.md) (amended
+below), [ADR 0012: Safe retries](0012-bounded-data-transfer-and-safe-retries.md) and
+[ADR 0013: Application-owned migrations](0013-application-owned-migrations.md).
+
+## Context
+
+Time-index tables on `timescale_db` DataSources grow without bound. Users need to
+compress older data and drop expired data on a schedule, without database access.
+
+Source inspection on 2026-10-02 found:
+
+- The Admin's Timescale Policies tab calls `GET/PATCH meta-tables/{uid}/policies`
+  and appears only when a table reports `capabilities.timescale_policies`. The API
+  provides neither, so the tab never appears.
+- No current path creates hypertables. `storage_layout.partitioning.strategy`
+  accepts `timescale_hypertable`, but only the legacy
+  `create_multi_index_table_for_metadata` calls `create_hypertable`. Timescale
+  policies require hypertables.
+- The legacy policy helpers in `adapters/maintenance/timescaledb.py` have no route
+  callers. They sleep between steps, remove and re-add policies in separate
+  commits, and some build SQL with f-strings.
+- TimescaleDB creates a policy job owned by the hypertable owner and refuses to
+  create it when that role lacks `LOGIN`. ADR 0007 makes `mt_owner`, a `NOLOGIN`
+  role, the owner of every managed table.
+
+## Decision
+
+### Core model
+
+**TimescaleDB stores the policy; MetaTables keeps no copy.** A policy is a
+Timescale background job. Timescale already schedules and runs it, so MetaTables
+adds no scheduler and no policy table. The API authorizes in the catalog, reads
+jobs from `timescaledb_information`, and writes with Timescale's policy functions.
+
+Policies are set only through the API and the Admin. An application that wants
+its policies in code runs a Job at bootstrap that calls the client method. Table
+contracts and application migrations carry no policy fields.
+
+| Policy | User sets | MetaTables derives |
+| --- | --- | --- |
+| Compression | `after`; optional `schedule_interval`, `initial_start`, `timezone` | `segmentby` = identity dimensions; `orderby` = time index `DESC` |
+| Retention | `after`; optional `schedule_interval`, `initial_start`, `timezone` | — |
+
+- `after: null` removes that policy. Removing compression leaves already
+  compressed chunks compressed.
+- If both are set, retention `after` must be longer than compression `after`.
+- Each change runs in one transaction with parameterized SQL and no sleeps.
+- Dropping a table removes its jobs.
+- TimescaleDB 2.11 or later is required, since it accepts inserts, updates,
+  deletes and upserts on compressed chunks. Uploads and tail deletes therefore
+  need no decompression step.
+
+### Eligible tables
+
+A table can have policies when:
+
+1. its DataSource is `timescale_db`;
+2. it is a time-index table and not `external_registered`;
+3. its physical table is a hypertable; and
+4. the installed TimescaleDB version is supported.
+
+Conditions 1–2 are catalog facts. They set `capabilities.timescale_policies` on
+the table response without opening a database connection. Conditions 3–4 are
+checked live when policies are read or written. Saving additionally requires a
+`read_write` DataSource. Tables on any other engine never show policies.
+
+### Hypertables
+
+On a `timescale_db` DataSource, managed finalization converts each new time-index
+table into a hypertable on its time index (`create_hypertable`, `if_not_exists`,
+`migrate_data`). Strategy `backend_default` means hypertable there; `none` opts
+out. `partitioning.strategy` is the only switch, so the duplicate
+`backend_hints.timescale.partitioning` hint and `time_index_table.partition_strategy`
+are removed. Chunk interval stays at Timescale's default. Every unique constraint
+must include the time index; the storage layout's `uniqueness` already does.
+
+### Authorization and ownership
+
+Writer on the table is the only permission check: a Writer can finalize the
+hypertable and set or remove both policies. Readers can view policies and job
+status. Organization admins also see every job on the DataSource.
+
+**Amendment to ADR 0007:** on `timescale_db` DataSources, hypertables are owned
+by the API's database login instead of `mt_owner`, so their policy jobs run as
+that login. Access setup leaves hypertable ownership unchanged instead of handing
+the tables to `mt_owner`. The API's login already owns the database and is a member of
+`mt_owner`, so this grants it nothing new. Reader/Writer grants, caller SQL roles,
+application migrations and other engines are unchanged. If the DataSource later
+uses a different login, existing jobs keep running as the old one until the
+tables are reassigned.
+
+Each policy change is recorded in the `physical_operation` journal as
+`timescale_policy`, with the requesting User. System migration
+`0009_timescale_policies` adds that kind, drops `time_index_table.partition_strategy`
+and removes the Timescale hint from stored layouts; existing runtimes upgrade
+through Settings.
+
+### API
+
+```text
+TimescalePolicy        { after, schedule_interval?, initial_start?, timezone? }
+TimescaleTablePolicies { compression, retention,
+                         eligibility { eligible, reason? },               read-only
+                         jobs [TimescaleJob],                             read-only
+                         compression_stats { before_bytes, after_bytes,
+                                             compressed_chunks, total_chunks } }  read-only
+TimescaleJob           { job_id, kind, table_uid?, hypertable, scheduled, status,
+                         last_run_status, last_successful_finish, next_start,
+                         total_failures, last_error? }
+```
+
+Intervals use PostgreSQL interval text such as `30 days`; the database validates them.
+
+| Endpoint | Access | Behavior |
+| --- | --- | --- |
+| `GET /meta-tables/{uid}/timescale-policies/` | Reader | Policies, eligibility, jobs and compression stats. |
+| `PUT /meta-tables/{uid}/timescale-policies/` | Writer | Replaces both policies; idempotent, safe to retry. Returns the GET body. |
+| `GET /data-sources/{uid}/timescale-jobs/` | Any User who can see the source | Jobs for tables the caller can read; admins see all. `timescale_db` only. |
+
+Errors: `404 timescale_policies_unavailable` for other engines,
+`403 write_access_required`, `409 timescale_not_hypertable`,
+`409 timescale_version_unsupported`, `409 timescale_extension_missing`, the existing
+DataSource access codes for read-only or disabled sources, and `422` for an invalid
+interval or timezone or `timescale_retention_not_after_compression`.
+
+The Python client exposes `get_timescale_policies()` and `set_timescale_policies()`
+on time-index tables.
+
+### Admin
+
+- **Table → Timescale Policies** (the existing tab, reworked): Compression and
+  Retention cards, each with an on/off switch, an `after` number with a unit
+  (days, weeks, months), and an Advanced section for schedule, start and timezone.
+  Each card shows its job's status (Scheduled, Running, Failed, Paused), last
+  success, next run and failures. Compression shows size before and after.
+  Saving retention asks for confirmation, naming the cutoff date and stating that
+  older chunks are permanently dropped on every run and that older backfills are
+  dropped too. Readers see the tab read-only. An ineligible table shows the reason.
+- **DataSource → Jobs** (`timescale_db` only): a table of jobs — table link,
+  policy, status, last run, next run, failures — with a failed-only filter and
+  Refresh, under Policies and Failed jobs counts.
+
+Screens use Command Center SDK components and theme tokens only.
+
+### Monitoring
+
+Monitoring is read on demand from `jobs`, `job_stats`, `job_errors` and
+`hypertable_compression_stats`. MetaTables stores no job history, polls nothing,
+and sends no alerts.
+
+## Consequences
+
+- One source of truth: a policy changed directly in the database appears in the
+  Admin as it is.
+- The legacy Timescale policy, decompression and hypertable helpers and their
+  runtime registrations are deleted.
+- Retention is destructive and available to every Writer; the confirmation is
+  the safeguard.
+- Policy jobs belong to the API's login. Rotating its password does not affect
+  them; no role gains `LOGIN`.
+
+## Alternatives considered
+
+- **Policies in table contracts or migrations.** Two authorities would overwrite
+  each other. A bootstrap Job calling the API covers code-defined policies.
+- **A catalog copy with a reconciler.** It duplicates Timescale's job store and
+  adds drift handling.
+- **A MetaTables scheduler.** Timescale already schedules policy jobs.
+- **`mt_owner` with `LOGIN` and no password.** It changes a login setting and
+  depends on the server never accepting passwordless authentication for that role.
+
+## Verification
+
+Focused tests cover eligibility, interval validation, authorization, the request
+contract and the derived compression settings, using a fake backend. The
+TimescaleDB container suite, run only on explicit request, verifies hypertable
+conversion, policy add/replace/remove, job ownership by the API's login, Reader and
+Writer grants reaching chunks and compressed chunks, and upserts into compressed
+chunks. Admin checks cover tab visibility, read-only mode and the retention
+confirmation.

{mainsequence_metatable-0.1.8.dev17 → mainsequence_metatable-0.1.9}/docs/adr/index.md

@@ -23,7 +23,7 @@ decisions belong in the SDK repository and are linked as dependencies.
 
 | ADR | Status | Scope |
 | --- | --- | --- |
-| [0001: One API execution path with SQLite for local development](api/0001-unified-api-storage-and-local-sqlite.md) | Accepted and implemented | Shared execution, API-owned local storage, SDK context dependency, and DuckDB removal; each runtime mode lists and selects only its own DataSources. |
+| [0001: One API execution path with SQLite for local development](api/0001-unified-api-storage-and-local-sqlite.md) | Accepted and implemented; hosted declaration verification pending | Shared execution, API-owned local storage, SDK context dependency, and DuckDB removal; the deployment declares the hosted runtime database in `configuration.yaml` and an Environment Secret, and Settings is read-only in Hosted mode. |
 | [0002: Application administration and table ownership](api/0002-application-administration-and-table-ownership.md) | Accepted and implemented | Platform admin facts, table Reader/Writer ownership, User/Team grants, live namespace inheritance, and application-owned access management. |
 | [0004: Execute approved application migrations inside the API](api/0004-approved-application-migrations.md) | Superseded by ADR 0013 | Historical API-installed provider execution model. |
 | [0005: Shared run logs with local file capture](api/0005-run-logs-and-local-file-capture.md) | Accepted and implemented; live hosted verification pending | Shared run correlation and log API, runtime-bound local JSONL files, and hosted retrieval through the SDK. |
@@ -34,4 +34,5 @@ decisions belong in the SDK repository and are linked as dependencies.
 | [0010: Import external tables and views](api/0010-import-external-tables-and-views.md) | Accepted; implemented (verification boundaries in ADR) | Shared backend discovery, metadata import/refresh, and bounded reads of external tables and views, preserving one runtime catalog and existing grants. |
 | [0012: Bounded data transfer, concurrent execution, and safe retries](api/0012-bounded-data-transfer-and-safe-retries.md) | Accepted; implemented; hosted engine verification pending | API/client row and byte budgets, incremental fetching, concurrent admission, durable chunk receipts, safe recovery, and required user documentation; independent of Artifacts and temporary URLs. |
 | [0013: Application-owned migrations](api/0013-application-owned-migrations.md) | Accepted and implemented | Application-side Alembic, environment database authority, API catalog integration and separate system migrations. |
-| [0014: System migrations in the deployment workflow](api/0014-main-sequence-release-jobs-and-production-migrations.md) | Accepted; implemented on development; hosted verification pending | A Main Sequence Job upgrades the active runtime from the candidate image before each API rollout. |
+| [0014: System migrations in the deployment workflow](api/0014-main-sequence-release-jobs-and-production-migrations.md) | Accepted; implemented on development; hosted verification pending | A Main Sequence Job verifies, migrates and registers the declared runtime database from the candidate image before each API rollout, including the first. |
+| [0015: TimescaleDB compression and retention policies](api/0015-timescale-compression-and-retention-policies.md) | Accepted; implemented on branch; TimescaleDB verification pending | Time-index tables on TimescaleDB become hypertables; Writers set compression and retention through the API/Admin; Timescale keeps the jobs, and the Admin shows their status. |