smart-data-engine-sdk 0.1.0.dev0__tar.gz → 0.1.0rc1__tar.gz

{smart_data_engine_sdk-0.1.0.dev0 → smart_data_engine_sdk-0.1.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: smart-data-engine-sdk
-Version: 0.1.0.dev0
+Version: 0.1.0rc1
 Summary: Smart Data Engine client library: declare a data model, we place it and move it
 Project-URL: Homepage, https://github.com/Smart-Data-Engines/smart-data-engine-sdk
 Project-URL: Repository, https://github.com/Smart-Data-Engines/smart-data-engine-sdk
@@ -17,11 +17,12 @@ Classifier: Intended Audience :: Developers
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Database
 Classifier: Typing :: Typed
-Requires-Python: <3.14,>=3.11
+Requires-Python: <3.15,>=3.11
 Provides-Extra: clickhouse
-Requires-Dist: clickhouse-connect>=0.7; extra == 'clickhouse'
+Requires-Dist: clickhouse-connect>=1.7.2; extra == 'clickhouse'
 Provides-Extra: dev
 Requires-Dist: mypy>=1.11; extra == 'dev'
 Requires-Dist: pytest>=8; extra == 'dev'
@@ -130,12 +131,9 @@ with zero releases — PyPI compares names after stripping `._-` and folding `l`
 `0`, and both reduce to `smartdataeng1ne`. The suffix changes the reduction, and it matches the
 repository name, which is the consistency worth having regardless.
 
-**`smart-data-engine-sdk` is unclaimed on PyPI, so those three commands install nothing today.** That
-is the honest state and not a typo: the library is not published yet. Read it as a warning rather
-than a footnote, because three error messages inside this library point at the same commands - if
-someone else registers the name before we do, a person who is already debugging gets told, by code
-they had decided to trust, to install a stranger's package.
-[`../docs/publishing.md`](../docs/publishing.md) is what closes this.
+Published to PyPI on 12 September 2026. Until then this section was the one instruction in this
+repository that was knowingly untrue, and it carried a warning saying so; the warning is gone because
+the commands work.
 
 ## Conformance
 
@@ -150,3 +148,27 @@ its own. If it is not, that is a bug in the document.
 ## Licence
 
 Apache-2.0.
+
+## Bound migration verification
+
+Managed cutover uses a `VerificationRequest` from the controller. Configure `Session(...,
+project_id=...)` from your local enrollment manifest, load the request with
+`VerificationRequest.from_record`, and pass it to `verify(..., request=request)`. The returned
+`as_record()` includes the request and counts, while row-level differences remain local.
+See [the request protocol](../docs/format-contract.md#7b-verification-requests-and-bound-reports).
+Loaded placement maps are immutable; load a new document when their layout or routing changes.
+
+Batch operations: [logical bulk writes](../docs/bulk-writes.md) document `save_many` / `saveMany`,
+bounds, source/copy ordering and uncertain outcomes.
+
+[Logical reads and exact summaries](../docs/logical-reads.md) cover scan, count and summarize,
+including portable keyset pages, typed values and explicit failure semantics.
+
+## Local Weather starter
+
+The [Weather starter](https://github.com/Smart-Data-Engines/smart-data-engine-sdk/blob/main/docs/weather-starter.md)
+is part of this package, as the `sde-weather` command. It exercises logical operations with
+restricted runtime credentials, and it includes local setup, telemetry, operator handoffs and an
+ownership-checked reset.
+
+Engine credentials and verified TLS configuration: [connection guide](../docs/engine-connections.md).

{smart_data_engine_sdk-0.1.0.dev0 → smart_data_engine_sdk-0.1.0rc1}/README.md

@@ -96,12 +96,9 @@ with zero releases — PyPI compares names after stripping `._-` and folding `l`
 `0`, and both reduce to `smartdataeng1ne`. The suffix changes the reduction, and it matches the
 repository name, which is the consistency worth having regardless.
 
-**`smart-data-engine-sdk` is unclaimed on PyPI, so those three commands install nothing today.** That
-is the honest state and not a typo: the library is not published yet. Read it as a warning rather
-than a footnote, because three error messages inside this library point at the same commands - if
-someone else registers the name before we do, a person who is already debugging gets told, by code
-they had decided to trust, to install a stranger's package.
-[`../docs/publishing.md`](../docs/publishing.md) is what closes this.
+Published to PyPI on 12 September 2026. Until then this section was the one instruction in this
+repository that was knowingly untrue, and it carried a warning saying so; the warning is gone because
+the commands work.
 
 ## Conformance
 
@@ -116,3 +113,27 @@ its own. If it is not, that is a bug in the document.
 ## Licence
 
 Apache-2.0.
+
+## Bound migration verification
+
+Managed cutover uses a `VerificationRequest` from the controller. Configure `Session(...,
+project_id=...)` from your local enrollment manifest, load the request with
+`VerificationRequest.from_record`, and pass it to `verify(..., request=request)`. The returned
+`as_record()` includes the request and counts, while row-level differences remain local.
+See [the request protocol](../docs/format-contract.md#7b-verification-requests-and-bound-reports).
+Loaded placement maps are immutable; load a new document when their layout or routing changes.
+
+Batch operations: [logical bulk writes](../docs/bulk-writes.md) document `save_many` / `saveMany`,
+bounds, source/copy ordering and uncertain outcomes.
+
+[Logical reads and exact summaries](../docs/logical-reads.md) cover scan, count and summarize,
+including portable keyset pages, typed values and explicit failure semantics.
+
+## Local Weather starter
+
+The [Weather starter](https://github.com/Smart-Data-Engines/smart-data-engine-sdk/blob/main/docs/weather-starter.md)
+is part of this package, as the `sde-weather` command. It exercises logical operations with
+restricted runtime credentials, and it includes local setup, telemetry, operator handoffs and an
+ownership-checked reset.
+
+Engine credentials and verified TLS configuration: [connection guide](../docs/engine-connections.md).

{smart_data_engine_sdk-0.1.0.dev0 → smart_data_engine_sdk-0.1.0rc1}/pyproject.toml

@@ -24,7 +24,7 @@ name = "smart-data-engine-sdk"
 dynamic = ["version"]
 description = "Smart Data Engine client library: declare a data model, we place it and move it"
 readme = "README.md"
-requires-python = ">=3.11,<3.14"
+requires-python = ">=3.11,<3.15"
 license = "Apache-2.0"
 # Apache-2.0 section 4 asks that the licence travel with the work, and section 4(d) says the same
 # about a NOTICE file where one exists. Both were missing from the wheel and the sdist until this
@@ -47,6 +47,7 @@ classifiers = [
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
     "Topic :: Database",
     # The wheel carries `py.typed`, so this row is a fact the artefact can be checked against.
     "Typing :: Typed",
@@ -68,13 +69,17 @@ Issues = "https://github.com/Smart-Data-Engines/smart-data-engine-sdk/issues"
 "Format contract" = "https://github.com/Smart-Data-Engines/smart-data-engine-sdk/blob/main/docs/format-contract.md"
 "Failure semantics" = "https://github.com/Smart-Data-Engines/smart-data-engine-sdk/blob/main/docs/failure-semantics.md"
 
+[project.scripts]
+sde-operator = "sde_operator.__main__:main"
+sde-weather = "sde_demo.__main__:main"
+
 [project.optional-dependencies]
 signed = ["cryptography>=42"]
 postgres = ["psycopg[binary]>=3.1"]
 # The official ClickHouse client, over HTTP. Chosen over the native-protocol driver because HTTP is
 # what a client is most likely to already have open through a proxy or a load balancer, and because
 # the second engine should not also introduce a second network story.
-clickhouse = ["clickhouse-connect>=0.7"]
+clickhouse = ["clickhouse-connect>=1.7.2"]
 dev = ["pytest>=8", "mypy>=1.11", "ruff>=0.6"]
 
 [tool.hatch.version]
@@ -84,7 +89,7 @@ path = "src/sde/__init__.py"
 # otherwise looks for a directory named after the project and fails - which it did, and better here
 # than at publish time.
 [tool.hatch.build.targets.wheel]
-packages = ["src/sde"]
+packages = ["src/sde", "src/sde_operator", "src/sde_demo"]
 # `src/sde/py.typed` needs no entry here: hatchling includes it because it lives inside the package
 # directory, which was checked by building the wheel both ways rather than assumed. A `force-include`
 # for it was written first and then removed - dead configuration that looks load-bearing is worse than

{smart_data_engine_sdk-0.1.0.dev0 → smart_data_engine_sdk-0.1.0rc1}/src/sde/__init__.py

@@ -33,22 +33,37 @@ mode, not a loophole.
 
 from __future__ import annotations
 
+from .bulk import MAX_BATCH_ROWS, MAX_BATCH_VALUES, BulkWritable
 from .canonical import CanonicalError, canonical_bytes, canonical_str, digest16
 from .capabilities import members_of, satisfies
+from .cutover import CUTOVER_PROTOCOL, CUTOVER_RELAYOUT_PROTOCOL, CutoverPlan, load_cutover_plan
 from .entity import Ref, clear_registry, entity, registry
 from .errors import (
+    BulkWriteRefused,
     DeclarationError,
     EngineError,
     MapError,
     MapRolledBack,
     MigrationRefused,
     ModelPlanningError,
+    ResourceBusy,
+    ResourceClosed,
     SdeError,
 )
 from .explain import Cost, Explains, PlanFinding, QueryPlan, QueryPlanRefused, explain
+from .frozen_verification import FrozenVerifyReport, verify_frozen
 from .groups import Group, colocation_groups, group_of
 from .hashing import NameMap, hash_identifiers, load_or_create_salt
+from .index_build import (
+    INDEX_CHANGE_PROTOCOL,
+    INDEX_PROTOCOL,
+    IndexPlan,
+    IndexReceipt,
+    index_build_name,
+    load_index_plan,
+)
 from .infer import InferredModel, Note, infer_model, infer_models
+from .inspection import InspectionContext
 from .internal import internal_failures, reset_internal_failures
 from .layout import (
     DIALECTS,
@@ -65,6 +80,7 @@ from .layout import (
     snake_case,
     stored_types,
 )
+from .local_cutover import CutoverReceipt, CutoverRecoveryRequired, LocalCutover, load_local_map
 from .migration import (
     BACKFILL_TABLE,
     CHUNK_ROWS,
@@ -80,6 +96,7 @@ from .migration import (
     verify,
 )
 from .model import CONTRACT, LogicalModel, build_model, neutral_declaration
+from .physical import PHYSICAL_DESIGN_SINCE, PhysicalFinding
 from .placement import (
     ALSO_WRITE_SINCE,
     MAP_CONTRACT,
@@ -91,10 +108,28 @@ from .placement import (
     PlacementMap,
     load_map,
 )
+from .provisioning import prepare_schema
+from .query import (
+    MAX_PAGE_ROWS,
+    NumericSummary,
+    Queryable,
+    QueryRefused,
+    Range,
+    ScanPage,
+    Summarizable,
+)
 from .routing import Router, resolve
 from .schema import CompatibilityViews, compatibility_views, schema_is_fixed, schema_statements
-from .session import Engine, Session
+from .session import Engine, ManagedEngine, Session
 from .shapes import SHAPE_KINDS, WRITE_KINDS, OperationShape, enumerate_shapes
+from .staging import (
+    STAGING_PROTOCOL,
+    STAGING_RELAYOUT_PROTOCOL,
+    StagingPlan,
+    StagingReceipt,
+    load_staging_plan,
+    staging_table_name,
+)
 from .telemetry import (
     MEASURED_FIELDS,
     CopyFreshness,
@@ -103,8 +138,12 @@ from .telemetry import (
     Histogram,
     Recorder,
     ShapeStats,
+    StorageMeasurement,
+    StorageSample,
+    StorageSize,
     Window,
     has_time_dimension,
+    time_fields,
 )
 from .types import Float32, Int32, Json, Timestamp, precision
 from .watermark import (
@@ -114,33 +153,51 @@ from .watermark import (
     WatermarkStore,
     enforce_forward_only,
 )
+from .write_fence import EPOCH_COLUMN as WRITE_EPOCH_COLUMN
+from .write_fence import FenceState, WriteFence
 
-__version__ = "0.1.0.dev0"
+__version__ = "0.1.0rc1"
 
 __all__ = [
     "ALSO_WRITE_SINCE",
     "BACKFILL_TABLE",
     "CHUNK_ROWS",
     "CONTRACT",
+    "CUTOVER_PROTOCOL",
+    "CUTOVER_RELAYOUT_PROTOCOL",
     "DIALECTS",
     "DIALECT_PRECISION",
     "FIXED_SCHEMA",
+    "INDEX_CHANGE_PROTOCOL",
+    "INDEX_PROTOCOL",
     "MAP_CONTRACT",
     "MAP_CONTRACT_FLOOR",
+    "MAX_BATCH_ROWS",
+    "MAX_BATCH_VALUES",
+    "MAX_PAGE_ROWS",
     "MEASURED_FIELDS",
     "ORDERBOOK_KEY",
     "ORDERBOOK_SHAPE",
     "ORDERBOOK_TABLE",
+    "PHYSICAL_DESIGN_SINCE",
     "PRECISION_INDEPENDENT",
     "RESERVED_TABLES",
     "SHAPE_KINDS",
+    "STAGING_PROTOCOL",
+    "STAGING_RELAYOUT_PROTOCOL",
     "WATERMARK_TABLE",
+    "WRITE_EPOCH_COLUMN",
     "WRITE_KINDS",
     "BackfillProgress",
+    "BulkWritable",
+    "BulkWriteRefused",
     "CanonicalError",
     "CompatibilityViews",
     "CopyFreshness",
     "Cost",
+    "CutoverPlan",
+    "CutoverReceipt",
+    "CutoverRecoveryRequired",
     "DeclarationError",
     "DerivedLayout",
     "Difference",
@@ -149,15 +206,22 @@ __all__ = [
     "EntityProgress",
     "Explains",
     "FanOutStats",
+    "FenceState",
     "Float32",
+    "FrozenVerifyReport",
     "Group",
     "GroupFeatures",
     "GroupPlacement",
     "Histogram",
+    "IndexPlan",
+    "IndexReceipt",
     "InferredModel",
+    "InspectionContext",
     "Int32",
     "Json",
+    "LocalCutover",
     "LogicalModel",
+    "ManagedEngine",
     "MapError",
     "MapRolledBack",
     "Materialization",
@@ -166,24 +230,40 @@ __all__ = [
     "ModelPlanningError",
     "NameMap",
     "Note",
+    "NumericSummary",
     "OperationShape",
+    "PhysicalFinding",
     "PhysicalLayout",
     "PlacementMap",
     "PlanFinding",
     "Protection",
     "QueryPlan",
     "QueryPlanRefused",
+    "QueryRefused",
+    "Queryable",
+    "Range",
     "Recorder",
     "Ref",
+    "ResourceBusy",
+    "ResourceClosed",
     "Router",
+    "ScanPage",
     "SdeError",
     "Session",
     "ShapeStats",
+    "StagingPlan",
+    "StagingReceipt",
+    "StorageMeasurement",
+    "StorageSample",
+    "StorageSize",
+    "Summarizable",
     "Timestamp",
+    "VerificationRequest",
     "VerifyReport",
     "WatermarkCheck",
     "WatermarkStore",
     "Window",
+    "WriteFence",
     "__version__",
     "backfill",
     "build_model",
@@ -205,15 +285,21 @@ __all__ = [
     "group_of",
     "has_time_dimension",
     "hash_identifiers",
+    "index_build_name",
     "infer_model",
     "infer_models",
     "internal_failures",
+    "load_cutover_plan",
+    "load_index_plan",
+    "load_local_map",
     "load_map",
     "load_or_create_salt",
+    "load_staging_plan",
     "members_of",
     "neutral_declaration",
     "precision",
     "precision_refusal",
+    "prepare_schema",
     "registry",
     "reset_internal_failures",
     "resolve",
@@ -221,6 +307,12 @@ __all__ = [
     "schema_is_fixed",
     "schema_statements",
     "snake_case",
+    "staging_table_name",
     "stored_types",
+    "time_fields",
+    "verification_request",
     "verify",
+    "verify_frozen",
 ]
+
+from .verification import VerificationRequest, verification_request

smart_data_engine_sdk-0.1.0rc1/src/sde/_cutover_project.py

@@ -0,0 +1,179 @@
+"""Durable project state for a client-side cutover; no rows or credentials are stored here."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from pathlib import Path
+from typing import Any
+
+from . import _local_state
+from .errors import MigrationRefused
+
+
+def encode(value: Any) -> bytes:
+    # Native principal/namespace names are exact strings, not canonical map identifiers.
+    # Keep their spelling; the state checksum is over this explicitly versioned raw-JSON form.
+    return json.dumps(
+        value, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False
+    ).encode("utf-8")
+
+
+def _abandoned_stages(payload: Any) -> bool:
+    stages = payload.get("stages") if isinstance(payload, dict) else None
+    if not isinstance(stages, dict):
+        return False
+    return any(
+        isinstance(record, dict)
+        and isinstance(record.get("receipt"), dict)
+        and record["receipt"].get("outcome") == "abandoned"
+        for record in stages.values()
+    )
+
+
+def _index_changes(payload: Any) -> bool:
+    """Whether an in-place index record - executing or kept - also removes indexes (protocol 2)."""
+    if not isinstance(payload, dict):
+        return False
+    records: list[Any] = []
+    history = payload.get("indexes")
+    if isinstance(history, dict):
+        records.extend(history.values())
+    execution = payload.get("execution")
+    if isinstance(execution, dict) and execution.get("kind") == "index":
+        records.append(execution)
+    return any(
+        isinstance(record, dict)
+        and isinstance(record.get("plan"), dict)
+        and record["plan"].get("protocol") == 2
+        for record in records
+    )
+
+
+class ProjectState:
+    """One POSIX directory and lock, shared by all local executors for this project."""
+
+    def __init__(self, root: Path, project_id: str, model_version: str) -> None:
+        self.root = root.resolve()
+        self.project_id = project_id
+        self.model_version = model_version
+        self.path = self.root / "project.json"
+        self.map_path = self.root / "active-map.json"
+
+    def lock(self) -> Any:
+        return _local_state.transaction(self.root)
+
+    def read(self) -> dict[str, Any]:
+        try:
+            envelope = json.loads(self.path.read_bytes())
+            if not isinstance(envelope, dict) or set(envelope) != {
+                "storage_contract",
+                "payload",
+                "sha256",
+            }:
+                raise ValueError("invalid state envelope")
+            if type(envelope["storage_contract"]) is not int or envelope[
+                "storage_contract"
+            ] not in (1, 2, 3, 4, 5):
+                raise ValueError("unsupported state storage contract")
+            payload = envelope["payload"]
+            if (
+                not isinstance(payload, dict)
+                or hashlib.sha256(encode(payload)).hexdigest() != envelope["sha256"]
+            ):
+                raise ValueError("state checksum mismatch")
+            if (
+                payload.get("project_id") != self.project_id
+                or payload.get("model_version") != self.model_version
+            ):
+                raise ValueError("state belongs to another local project/model")
+            fields = {
+                "project_id",
+                "model_version",
+                "active_map",
+                "execution",
+                "completed",
+                "retired_names",
+            }
+            if envelope["storage_contract"] >= 2:
+                fields.add("stages")
+                if not isinstance(payload.get("stages"), dict):
+                    raise ValueError("staging history must be an object")
+            if envelope["storage_contract"] >= 3:
+                # In-place index builds: a reader of contracts 1 and 2 refuses this state rather
+                # than ignoring a history it would not know to keep.
+                fields.add("indexes")
+                if not isinstance(payload.get("indexes"), dict):
+                    raise ValueError("index build history must be an object")
+            if envelope["storage_contract"] < 4 and _abandoned_stages(payload):
+                # Contract 4 is what makes an older operator refuse an abandoned staging's record
+                # instead of reading a receipt whose tables may name no identity.
+                raise ValueError("an abandoned staging needs state storage contract 4")
+            if envelope["storage_contract"] < 5 and _index_changes(payload):
+                # Contract 5 is what makes an older operator refuse a build that also removes
+                # indexes, instead of resuming it with no idea that removals follow the decision.
+                raise ValueError("an index change needs state storage contract 5")
+            if set(payload) != fields:
+                raise ValueError("unknown or missing project state fields")
+            return payload
+        except (OSError, ValueError, TypeError) as exc:
+            raise MigrationRefused(
+                "local cutover state is missing or corrupt; "
+                "restore verified state before proceeding"
+            ) from exc
+
+    def confirm(self) -> None:
+        # Call under the project lock. Neither reading matching bytes nor an in-memory receipt
+        # confirms the directory fsync that may have failed after an earlier publication.
+        _local_state.confirm_file(self.map_path)
+        _local_state.confirm_file(self.path)
+
+    def write(self, payload: dict[str, Any]) -> None:
+        body = encode(payload)
+        envelope = {
+            "storage_contract": 5
+            if _index_changes(payload)
+            else 4
+            if _abandoned_stages(payload)
+            else 3
+            if "indexes" in payload
+            else 2
+            if "stages" in payload
+            else 1,
+            "payload": payload,
+            "sha256": hashlib.sha256(body).hexdigest(),
+        }
+        _local_state.write_bytes(self.path, encode(envelope))
+
+    def enroll(self, document: dict[str, Any], map_payload: bytes) -> None:
+        with self.lock():
+            if self.map_path.exists() and self.map_path.read_bytes() != map_payload:
+                raise MigrationRefused("an existing active map differs from the enrolled state")
+            if self.path.exists():
+                state = self.read()
+                if (
+                    state["active_map"] != document
+                    or state["execution"] is not None
+                    or state["completed"]
+                ):
+                    raise MigrationRefused(
+                        "local project is already enrolled; use its current state"
+                    )
+            else:
+                state = {
+                    "project_id": self.project_id,
+                    "model_version": self.model_version,
+                    "active_map": document,
+                    "execution": None,
+                    "completed": {},
+                    "retired_names": [],
+                }
+                self.write(state)
+            if not self.map_path.exists():
+                _local_state.write_bytes(self.map_path, map_payload, replace=False)
+            # Matching visible files may come from a publication whose directory fsync failed.
+            # An identical retry must confirm their durability before enrollment succeeds.
+            self.confirm()
+
+    def publish(self, payload: bytes) -> None:
+        _local_state.write_bytes(self.map_path, payload)