mainsequence-metatable 0.1.3__tar.gz → 0.1.4.dev5__tar.gz

mainsequence_metatable-0.1.4.dev5/.agents/skills/client/metatables-local-development/SKILL.md

@@ -0,0 +1,165 @@
+---
+name: metatables-local-development
+description: "Develop and test applications using the installed MetaTables client against local SQLite before returning to the intended environment database. Covers project/API/Admin setup, runtime verification, isolated fixtures, approved migrations and switching back after verification. Excludes MetaTables client/API implementation and hosted frontend deployment."
+---
+
+# Local-first MetaTables application development
+
+Use local SQLite by default when building or changing a consuming application's
+tables, migration providers, queries or producers. Exercise the application
+through the same MetaTables API/client used with the environment database. Keep
+development fixtures and test writes in the local runtime, then return to the
+intended environment as part of the requested workflow. Respect an explicitly
+requested target or backend-specific test scope.
+
+In a copied skill, resolve the `docs/` and `examples/` paths below relative to
+this skill's `references/` directory. In the MetaTables source checkout, resolve
+them from the repository root. Read `docs/operations/local-runtime.md` for runtime
+selection and persistence, and `docs/client/installation-and-connection.md` when
+installing or troubleshooting. `examples/local_app/README.md` provides a complete
+installed-package migration/write/read example.
+
+## Select local storage deliberately
+
+Three controls have different meanings:
+
+- `local_mode_available: true` permits local development; it does not select the
+  current database. `metatables init --local` prepares this configuration.
+- `metatables serve --local` starts the API in Local mode. Add `--admin` to run
+  the existing Vite development frontend. Admin Settings can subsequently switch
+  that running API to Hosted without changing its loopback address.
+- Global `metatables --local ...` and `configure_local_client()` select the
+  project's running API connection. They do not force its database back to Local.
+
+Before migrations or a batch of test writes, inspect the API's actual runtime.
+A loopback URL, a local token, a test namespace or an updater hash is not proof of
+storage isolation. Require `local_mode: true`; after initialization also require
+`dialect: sqlite` and the expected local runtime DataSource. If these disagree,
+stop the mutating work and select Local in Settings or restart with `serve --local`.
+Do not retry against automatic hosted discovery when a local connection fails.
+
+## Establish the development loop
+
+1. Work in the consuming application's Git checkout, with its commit and origin
+   remote. Use the Python environment containing MetaTables and the application's
+   installed provider code. Reuse its existing configuration and SDK login.
+2. For initial setup, run:
+
+   ```bash
+   mainsequence login
+   metatables init --local
+   metatables serve --local --admin
+   ```
+
+   Omit `--admin` for CLI-only work. The API is included in the Python package;
+   a MetaTables source checkout is unnecessary. Admin runs through Vite. Its
+   pinned source downloads only when missing, and ready dependencies are reused.
+   Use `--admin-path` for an already prepared checkout. Follow the installation
+   guide for Node/npm requirements and repository access; do not clone on each run.
+3. In another terminal in the same project, inspect `metatables --local runtime
+   status`. Confirm Local and review the selected SQLite candidate before any
+   initialization. If system migrations are needed, explicitly run
+   `metatables --local runtime initialize`, then inspect status again.
+   Starting the API does not initialize or migrate a database.
+4. Author and review application migrations using the
+   [migrations skill](../metatables-migrations/SKILL.md). Approve the installed
+   provider in configuration and apply its head to the verified local runtime:
+
+   ```bash
+   metatables init --local --provider ledger=ledger.migrations:migration
+   metatables --local migrations upgrade --provider ledger
+   ```
+
+   Substitute the application's own alias and module. Execution takes an approved
+   alias; authoring takes a Python reference. System initialization and application
+   migration history remain separate.
+5. Build application contracts/queries with the
+   [table skill](../metatables-meta-tables/SKILL.md), or producers/readers with the
+   [updater skill](../metatables-time-index-table-updates/SKILL.md). Use small,
+   deterministic local inputs, run the relevant application checks, and inspect
+   actual rows and update results through the API.
+
+## Keep verification local and reproducible
+
+Make local integration-test setup select the connection before constructing
+clients, resolving table bindings, or creating updaters. Fail before writes when
+the selected runtime is wrong. A consuming application's test setup can use:
+
+```python
+from metatables import configure_local_client, get_runtime_status
+
+configure_local_client()
+state = get_runtime_status()
+source = state.get("data_source") or {}
+bootstrap = state.get("bootstrap") or {}
+if not (
+    state.get("local_mode") is True
+    and state.get("dialect") == "sqlite"
+    and source.get("class_type") == "sqlite"
+    and source.get("uid")
+    and bootstrap.get("active") is True
+    and bootstrap.get("status") == "ready"
+    and not state.get("data_source_error")
+):
+    raise RuntimeError("Tests require an initialized local SQLite runtime.")
+local_source_uid = source["uid"]
+```
+
+Resolve each test output in that runtime and check its DataSource UID against
+`local_source_uid`. Keep fixtures, migration providers and writable dependencies
+there too. Local mode does not isolate unrelated HTTP calls, external connections,
+or writes explicitly targeting another registered DataSource: replace those
+side effects with fixtures or controlled test dependencies. SDK identity checks
+still use the configured platform session.
+
+Run pure contract/frame checks first, then the smallest useful API integration
+case: apply the reviewed provider, seed a small fixture, execute the query or
+producer, read and assert its output, and repeat to verify the intended incremental
+or idempotent behavior. For a persistence claim, restart and read existing rows
+before seeding again. Recheck runtime state after any restart or mode/source change;
+do not switch modes while tests or producers are running.
+
+Local storage persists across launches and Git branches of the checkout. A new
+branch does not create a fresh test database. Use identifiable fixtures and clean
+up only the rows owned by the test. When a fresh database is necessary, select a
+separate temporary SQLite file through Settings and initialize it explicitly;
+restore the previous selection afterwards. Do not delete or replace a developer's
+existing database just to obtain a clean test run.
+
+SQLite verifies portable application behavior, not PostgreSQL/MySQL/MSSQL-specific
+SQL, Timescale features, database roles or every concurrency behavior. Report that
+limit and use the intended engine in an isolated test database when the task
+requires those checks. Do not use the shared environment database as the default
+integration-test fixture.
+
+## Finish verification and switch back
+
+Record what passed, the runtime/DataSource used, and any engine-specific work still
+unverified. Carry forward the reviewed application code and migration revisions;
+local rows, catalog UIDs, credentials and fixture data are not promoted by a switch.
+
+When returning to the environment is part of the requested workflow:
+
+1. Finish local tests and stop active writers. In Admin **Settings → Runtime mode**,
+   inspect the displayed hosted environment and choose the intended registered
+   DataSource. Use **Switch to Hosted**. This changes the API worker and selected
+   database; it does not copy or merge the local database. Leave local capability
+   enabled if the developer needs to switch back later.
+2. Re-read runtime status and verify `local_mode: false`, the intended verified
+   hosted environment, the expected DataSource and readiness. The same local
+   connection command still reaches this supervised API; its `--local` flag alone
+   does not prove SQLite. If the switch fails and the old worker is restored,
+   report the actual mode instead of claiming the switch succeeded.
+3. Restart application/test client processes and resolve catalog bindings again.
+   Do not reuse local table UIDs or already-bound updater instances against the
+   environment database. Apply reviewed revisions or perform environment writes
+   only within the user's requested scope; passing local tests is not itself a
+   request to seed fixtures, run backfills or migrate that database.
+
+If the user instead wants to connect to a separately deployed API, follow the
+installation guide's hosted endpoint selection. Restore prior connection settings
+and start a fresh client process without `configure_local_client()`; remove only
+transport overrides introduced for local work. Do not redirect the SDK platform
+endpoint, invent `serve --hosted`, or use an environment-variable runtime toggle.
+A task limited to local development can finish with verified local results and
+clear remaining steps, without switching the runtime or modifying hosted data.

{mainsequence_metatable-0.1.3 → mainsequence_metatable-0.1.4.dev5}/.agents/skills/client/metatables-meta-tables/SKILL.md

@@ -17,6 +17,11 @@ this skill's `references/` directory. The client CLI bundles the matching versio
 guides and examples there, including their linked documents. In the MetaTables
 source checkout, read the same paths from the repository root.
 
+For application development and mutation tests, use the
+[local development skill](../metatables-local-development/SKILL.md) to select and
+verify local SQLite before writing, then return to the intended environment after
+verification within the requested scope.
+
 Use the installed `metatables` client. The API catalog resource, the client
 resource model, and a SQLAlchemy authoring class have different responsibilities.
 Read `docs/concepts/architecture.md` and
@@ -28,7 +33,8 @@ identity and Git source facts use SDK imports. Table operations use the client's
 `METATABLES_API_URL`; the API selects the effective DataSource. See
 `docs/client/installation-and-connection.md`. Local development runs the same API
 with SQLite behind it; ordinary clients never open local table files. Local
-workspaces use one SQLite database for system and user tables per checkout/branch.
+workspaces use one persistent SQLite database for system and user tables per
+checkout; Git branches share it.
 Settings explicitly initializes or selects the runtime DataSource in both modes;
 startup never runs Alembic. See `docs/operations/catalog-migrations.md`.
 
@@ -104,3 +110,19 @@ against the actual workflow. Run relevant behavior tests and the example tests.
 State whether validation used local contracts, a disposable database, or a
 configured service. Never claim an end-to-end deployment result from an import
 check. Preserve published contracts unless the task authorizes their evolution.
+
+## Bounded transfers and recovery
+
+Read `docs/client/bounded-transfers.md` for transfer contract v1. Prefer
+`MetaTable.iter_rows()` for structured relation batches and
+`TimeIndexTableRef.iter_batches()` for typed time-index frames. Existing complete
+DataFrame readers have row, serialized-byte, and elapsed-time budgets; increase
+client collection budgets deliberately or narrow the selection. Offset pages do
+not provide a snapshot, and iterators do not prefetch.
+
+Uploads remain sequential, with at most 10,000 rows and byte checks per chunk.
+Each chunk commits independently. Preserve an explicit operation UUID and input
+for restart recovery. On an uncertain outcome, inspect `upload_receipt()` before
+resubmitting; do not assign a fresh key to bypass uncertainty. Administrator
+reconciliation requires verified physical evidence. This workflow uses the Main
+Sequence API and has no Artifact or temporary-URL dependency.

{mainsequence_metatable-0.1.3 → mainsequence_metatable-0.1.4.dev5}/.agents/skills/client/metatables-migrations/SKILL.md

@@ -17,6 +17,11 @@ this skill's `references/` directory. The client CLI bundles the matching versio
 guides and examples there, including their linked documents. In the MetaTables
 source checkout, read the same paths from the repository root.
 
+For application development and mutation tests, use the
+[local development skill](../metatables-local-development/SKILL.md) to select and
+verify local SQLite before writing, then return to the intended environment after
+verification within the requested scope.
+
 Read `docs/client/define-and-migrate-tables.md`,
 `docs/concepts/table-contracts-and-lifecycle.md`, and the tested
 `examples/tables.py`. Managed authoring is migration-first: define SQLAlchemy
@@ -76,9 +81,18 @@ revision for every later schema change.
 
 ## Execute through the approved runtime
 
-Client-side `current`, `upgrade`, `downgrade`, direct migration connections, and
-database autogeneration are unsupported. `--sqlalchemy-url` does not bypass that
-boundary.
+Direct client-side Alembic execution, `current`, `downgrade`, migration
+connections and database autogeneration are unsupported. `--sqlalchemy-url` does
+not bypass that boundary. The supported upgrade command executes inside the API.
+After verifying Local mode and the selected DataSource, use:
+
+```bash
+metatables --local migrations upgrade --provider ledger
+```
+
+`ledger` is the approved alias, not a Python provider reference. Only `head` is
+supported. The global `--local` selects the running project's API connection;
+inspect runtime status because Admin can switch that API to Hosted.
 
 The API applies a provider through `POST /application-migrations/upgrade/`, which
 accepts only an alias listed in its `application_migration_providers`. The

{mainsequence_metatable-0.1.3 → mainsequence_metatable-0.1.4.dev5}/.agents/skills/client/metatables-time-index-table-updates/SKILL.md

@@ -17,6 +17,11 @@ this skill's `references/` directory. The client CLI bundles the matching versio
 guides and examples there, including their linked documents. In the MetaTables
 source checkout, read the same paths from the repository root.
 
+For application development and mutation tests, use the
+[local development skill](../metatables-local-development/SKILL.md) to select and
+verify local SQLite before writing, then return to the intended environment after
+verification within the requested scope.
+
 Read `docs/client/build-an-updater.md`,
 `docs/concepts/time-index-tables-and-updates.md`, and the tested
 `examples/updater.py`. Use `docs/client/read-existing-time-index-tables.md` and
@@ -101,3 +106,19 @@ actually exercised: pure frame logic, disposable storage, or a configured API.
 Do not claim a scheduling/release workflow from producer tests. If a published
 contract decision remains unknown, resolve that decision while continuing any
 independent authorized work.
+
+## Bounded transfers and recovery
+
+Read `docs/client/bounded-transfers.md` for transfer contract v1. Prefer
+`MetaTable.iter_rows()` for structured relation batches and
+`TimeIndexTableRef.iter_batches()` for typed time-index frames. Existing complete
+DataFrame readers have row, serialized-byte, and elapsed-time budgets; increase
+client collection budgets deliberately or narrow the selection. Offset pages do
+not provide a snapshot, and iterators do not prefetch.
+
+Uploads remain sequential, with at most 10,000 rows and byte checks per chunk.
+Each chunk commits independently. Preserve an explicit operation UUID and input
+for restart recovery. On an uncertain outcome, inspect `upload_receipt()` before
+resubmitting; do not assign a fresh key to bypass uncertainty. Administrator
+reconciliation requires verified physical evidence. This workflow uses the Main
+Sequence API and has no Artifact or temporary-URL dependency.

{mainsequence_metatable-0.1.3/mainsequence_metatable.egg-info → mainsequence_metatable-0.1.4.dev5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.3
+Version: 0.1.4.dev5
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT
@@ -14,7 +14,7 @@ Classifier: Operating System :: OS Independent
 Requires-Python: >=3.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: mainsequence[server]<9,>=8.1.25
+Requires-Dist: mainsequence[server]<10,>=9.0.1
 Requires-Dist: alembic<2,>=1.18
 Requires-Dist: cachetools
 Requires-Dist: cryptography<51,>=46
@@ -27,13 +27,14 @@ Requires-Dist: opentelemetry-exporter-otlp
 Requires-Dist: opentelemetry-sdk
 Requires-Dist: pandas
 Requires-Dist: psutil
+Requires-Dist: platformdirs<5,>=4
 Requires-Dist: psycopg2-binary<3,>=2.9
 Requires-Dist: pyodbc<6,>=5
 Requires-Dist: pydantic<3,>=2.11
 Requires-Dist: pyyaml
 Requires-Dist: requests>=2.33.0
 Requires-Dist: sqlalchemy<3,>=2.1
-Requires-Dist: psycopg[binary]<4,>=3.2
+Requires-Dist: psycopg[binary]==3.3.6
 Requires-Dist: structlog
 Requires-Dist: tqdm<5,>=4.66
 Requires-Dist: typer<1,>=0.12
@@ -86,7 +87,7 @@ through Settings; startup never runs Alembic.
 
 ## Start here
 
-- [Install and connect](docs/client/installation-and-connection.md), including the current local SDK prerequisite.
+- [Install and connect](docs/client/installation-and-connection.md), including the complete local CLI workflow.
 - [Understand the concepts](docs/concepts/architecture.md) and [identity and scope](docs/concepts/identity-and-scope.md).
 - [Understand the security model](docs/security/index.md): administration, Reader/Writer table ownership, sharing, and implementation status.
 - [Follow the market-data tutorial](docs/client/tutorial.md) with its complete examples folder.
@@ -109,6 +110,10 @@ Copy the installed client's skills into a consuming application explicitly:
 metatables copy-metatables-skills --path /path/to/application
 ```
 
+The local development skill guides application work through local SQLite setup,
+verified test writes, and an explicit switch back to the intended environment.
+The other skills cover table contracts, application migrations and producers.
+
 The command uses the Main Sequence SDK helper, writes to
 `.agents/skills/metatables/`, and records the client version. It works without
 login or an API. Use `--dry-run` to preview the copy, or put global `--json` before
@@ -117,7 +122,7 @@ for refresh behavior and the bundled references.
 
 ## Development
 
-Python 3.13 or newer is required. After installing the compatible local SDK:
+Python 3.13 or newer is required. The package selects the published compatible SDK:
 
 ```bash
 python -m pip install -e '.[dev,docs]'

{mainsequence_metatable-0.1.3 → mainsequence_metatable-0.1.4.dev5}/README.md

@@ -32,7 +32,7 @@ through Settings; startup never runs Alembic.
 
 ## Start here
 
-- [Install and connect](docs/client/installation-and-connection.md), including the current local SDK prerequisite.
+- [Install and connect](docs/client/installation-and-connection.md), including the complete local CLI workflow.
 - [Understand the concepts](docs/concepts/architecture.md) and [identity and scope](docs/concepts/identity-and-scope.md).
 - [Understand the security model](docs/security/index.md): administration, Reader/Writer table ownership, sharing, and implementation status.
 - [Follow the market-data tutorial](docs/client/tutorial.md) with its complete examples folder.
@@ -55,6 +55,10 @@ Copy the installed client's skills into a consuming application explicitly:
 metatables copy-metatables-skills --path /path/to/application
 ```
 
+The local development skill guides application work through local SQLite setup,
+verified test writes, and an explicit switch back to the intended environment.
+The other skills cover table contracts, application migrations and producers.
+
 The command uses the Main Sequence SDK helper, writes to
 `.agents/skills/metatables/`, and records the client version. It works without
 login or an API. Use `--dry-run` to preview the copy, or put global `--json` before
@@ -63,7 +67,7 @@ for refresh behavior and the bundled references.
 
 ## Development
 
-Python 3.13 or newer is required. After installing the compatible local SDK:
+Python 3.13 or newer is required. The package selects the published compatible SDK:
 
 ```bash
 python -m pip install -e '.[dev,docs]'

{mainsequence_metatable-0.1.3 → mainsequence_metatable-0.1.4.dev5}/api/app/development.py

@@ -52,7 +52,11 @@ class DevelopmentSupervisor:
         self.previous = None
         self.ready = False
         self.rolling_back = False
-        self.start(self.mode)
+        try:
+            self.start(self.mode)
+        except BaseException:
+            self.close()
+            raise
 
     def start(self, mode, *, error=None):
         DeploymentConfiguration.load(self.configuration).require_development()

mainsequence_metatable-0.1.4.dev5/api/app/development_client.py

@@ -0,0 +1,10 @@
+"""Compatibility imports for existing local example/debug launchers."""
+
+from metatables.local_connection import (
+    CLIENT_FILE,
+    clear_development_client,
+    configure_development_client,
+    publish_development_client,
+)
+
+__all__ = ["CLIENT_FILE", "clear_development_client", "configure_development_client", "publish_development_client"]

{mainsequence_metatable-0.1.3 → mainsequence_metatable-0.1.4.dev5}/api/app/main.py

@@ -124,6 +124,16 @@ def create_app(config: RuntimeConfig | None = None) -> FastAPI:
         } or error.code.endswith("outcome_unknown")
         return JSONResponse(status_code=503 if unavailable else 400,
                             content={"detail": {"code": error.code, "detail": error.detail}})
+    from api.app.transfer import TransferMiddleware
+    from metatables.transfer_contract import TransferDeadlineError, TransferError
+
+    service.add_middleware(TransferMiddleware)
+
+    @service.exception_handler(TransferError)
+    async def transfer_error(request, error):
+        return JSONResponse(status_code=408 if isinstance(error, TransferDeadlineError) else 413,
+                            content={"detail": {"code": str(error)}})
+
     service.state.runtime = RuntimeState(config=selected)
     from api.app.runtime_control import RuntimeControl
     service.state.runtime_control = RuntimeControl(None, selected.configuration_path, None)

{mainsequence_metatable-0.1.3 → mainsequence_metatable-0.1.4.dev5}/api/app/routes/runtime_context.py

@@ -54,10 +54,11 @@ def developer_hosted_environment(cache: PlatformCache[UUID, HostedEnvironment])
         return HostedEnvironment(uid=None, status="unavailable")
 
 
-@router.get("/runtime-context/", response_model=RuntimeContext)
+@router.get("/runtime-context/", response_model=RuntimeContext, response_model_exclude_unset=True)
 def runtime_context(
     request: Request, response: Response,
     caller: AdmittedCaller = Depends(admit_request),
+    transfer_contract: int | None = None,
 ):
     state = request.app.state.runtime
     from api.app.configuration import DeploymentConfiguration
@@ -69,6 +70,8 @@ def runtime_context(
     environment_target = environment
     if state.config.local_mode and state.config.developer_mode and deployment.local_mode_available and control is not None and control.shutdown is not None:
         environment_target = developer_hosted_environment(state.environment_cache)
+    from metatables.transfer_contract import DEFAULT_LIMITS
+
     result = {
         "credential_store": state.credential_store.descriptor(),
         "application_migrations_available": bool(getattr(deployment, "application_migration_providers", {})),
@@ -90,6 +93,8 @@ def runtime_context(
         "data_source": None, "dialect": None, "paramstyle": None,
         "default_schema": None,
     }
+    if transfer_contract == 1:
+        result["transfer_limits"] = DEFAULT_LIMITS.model_dump()
     response.headers["Cache-Control"] = "no-store"
     if state.bootstrap is not None and not state.bootstrap.active:
         result["data_source_error"] = "runtime_not_initialized"

{mainsequence_metatable-0.1.3 → mainsequence_metatable-0.1.4.dev5}/api/app/routes/time_index_delete.py

@@ -29,7 +29,6 @@ from api.backend.domain.permission_rules import TrustedActorFacts
 from api.backend.integrations.data_source_provider import DataSourceProvider
 from api.backend.integrations.runtime_data_sources import DataSourceIntegrationError
 from api.backend.persistence.grants import GrantRepository
-from api.backend.persistence.locking import lock_catalog
 from api.backend.persistence.models import MetaTable, MetaTableColumn, TimeIndexTable
 
 router = APIRouter(prefix="/time-index-meta-tables", tags=["TimeIndexMetaTables"])
@@ -64,9 +63,8 @@ def delete_after_date(
     )
     if payload.after_date is None and not scoped:
         raise HTTPException(status_code=422, detail="after_date or a dimension scope is required.")
-    lock_catalog(session)
-    table = session.get(MetaTable, table_uid, with_for_update=True)
-    specialization = session.get(TimeIndexTable, table_uid, with_for_update=True)
+    table = session.get(MetaTable, table_uid)
+    specialization = session.get(TimeIndexTable, table_uid)
     if (
         table is None or specialization is None or table.provisioning_status != 'active' or (not GrantRepository(session).may_access_table(table_uid=table_uid, actor=actor, edit=True))
     ):

{mainsequence_metatable-0.1.3 → mainsequence_metatable-0.1.4.dev5}/api/app/routes/time_index_reads.py

@@ -45,7 +45,7 @@ class RangeRead(DimensionSelection):
     less_or_equal: bool | None = True
     columns: list[str] | None = Field(default=None, max_length=128)
     offset: int = Field(default=0, ge=0, le=1_000_000)
-    limit: int = Field(default=500, ge=1, le=14_000)
+    limit: int = Field(default=500, ge=1, le=10_000)
     table_update_uid: UUID | None = None
 
 
@@ -87,7 +87,7 @@ def _read(
             table.data_source_uid,
         )
         from api.backend.backends.operations import caller_table_source
-        source = caller_table_source(source, session, actor)
+        source = caller_table_source(source, session, actor, table_uid=table_uid)
         params: dict[str, Any] = {
             "source": source,
             "physical_schema": table.physical_schema,