supertable 2.4.0__tar.gz → 2.5.3__tar.gz

supertable-2.5.3/MANIFEST.in

@@ -0,0 +1,8 @@
+include LICENSE
+include README.md
+include requirements.txt
+
+# The published source archive is an installable runtime artifact.  Repository
+# release-gate tests depend on workflow and baseline files that are deliberately
+# not shipped, so do not include a misleading, incomplete top-level test tree.
+prune tests

{supertable-2.4.0/supertable.egg-info → supertable-2.5.3}/PKG-INFO

@@ -1,51 +1,60 @@
 Metadata-Version: 2.4
 Name: supertable
-Version: 2.4.0
+Version: 2.5.3
 Summary: SuperTable — versioned data lake library for SQL analytics on Parquet + Redis.
-Author: Levente Kupas
 Author-email: Levente Kupas <lkupas@kladnasoft.com>
-License: Super Table Public Use License (STPUL) v1.0
+License-Expression: LicenseRef-FSL-1.1-ALv2
 Project-URL: Homepage, https://github.com/kladnasoft/supertable
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: License :: Other/Proprietary License
-Classifier: Operating System :: OS Independent
+Classifier: Operating System :: POSIX
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy<3.0,>=1.26
 Requires-Dist: pandas<2.3,>=2.1
-Requires-Dist: pyarrow<19,>=16
+Requires-Dist: pyarrow<24,>=23.0.1
 Requires-Dist: pytz>=2023.3
 Requires-Dist: tzdata>=2023.3
-Requires-Dist: duckdb<2.0,>=1.1
-Requires-Dist: polars<2.0,>=1.6
-Requires-Dist: sqlglot<27,>=26
+Requires-Dist: apscheduler<4.0,>=3.11.3
+Requires-Dist: duckdb<2.0,>=1.5.1
+Requires-Dist: polars<2.0,>=1.36.1
+Requires-Dist: sqlglot<27,>=26.33
 Requires-Dist: colorlog<7.0,>=6.9
-Requires-Dist: python-dotenv<2.0,>=1.0
-Requires-Dist: setuptools>=68
+Requires-Dist: tqdm<5.0,>=4.66.3
+Requires-Dist: python-dotenv<2.0,>=1.2.2
+Requires-Dist: starlette<2.0,>=1.3.1
+Requires-Dist: httpx<1.0,>=0.27.1
+Requires-Dist: cryptography<51,>=50.0.0
 Requires-Dist: redis<6.0,>=5.0
 Provides-Extra: s3
-Requires-Dist: boto3<2.0,>=1.34; extra == "s3"
+Requires-Dist: boto3<2.0,>=1.35.2; extra == "s3"
 Provides-Extra: minio
 Requires-Dist: minio<8.0,>=7.2; extra == "minio"
 Provides-Extra: azure
-Requires-Dist: azure-storage-blob>=12.26.0; extra == "azure"
+Requires-Dist: azure-storage-blob<13.0,>=12.26.0; extra == "azure"
+Requires-Dist: azure-identity<2.0,>=1.17; extra == "azure"
 Provides-Extra: gcp
-Requires-Dist: google-cloud-storage>=3.1.0; extra == "gcp"
+Requires-Dist: google-cloud-storage<4.0,>=3.1.0; extra == "gcp"
 Provides-Extra: all-cloud
-Requires-Dist: supertable[azure,gcp,minio,s3]; extra == "all-cloud"
+Requires-Dist: boto3<2.0,>=1.35.2; extra == "all-cloud"
+Requires-Dist: minio<8.0,>=7.2; extra == "all-cloud"
+Requires-Dist: azure-storage-blob<13.0,>=12.26.0; extra == "all-cloud"
+Requires-Dist: azure-identity<2.0,>=1.17; extra == "all-cloud"
+Requires-Dist: google-cloud-storage<4.0,>=3.1.0; extra == "all-cloud"
 Provides-Extra: all
-Requires-Dist: supertable[all-cloud]; extra == "all"
-Dynamic: author
+Requires-Dist: boto3<2.0,>=1.35.2; extra == "all"
+Requires-Dist: minio<8.0,>=7.2; extra == "all"
+Requires-Dist: azure-storage-blob<13.0,>=12.26.0; extra == "all"
+Requires-Dist: azure-identity<2.0,>=1.17; extra == "all"
+Requires-Dist: google-cloud-storage<4.0,>=3.1.0; extra == "all"
 Dynamic: license-file
-Dynamic: requires-python
 
 # SuperTable
 
 ![Python](https://img.shields.io/badge/python-3.10%2B-blue)
 ![License](https://img.shields.io/badge/license-FSL--1.1--ALv2-orange)
-![Version](https://img.shields.io/badge/version-2.0.0-brightgreen)
+![Version](https://img.shields.io/badge/version-2.5.1-brightgreen)
 
 **SuperTable — versioned data lake library for SQL analytics.**
 
@@ -66,8 +75,9 @@ pip install "supertable[gcp]"         # Google Cloud Storage
 pip install "supertable[all]"         # everything
 ```
 
-Requirements: Python 3.10+, a reachable Redis 6+, and a configured storage
-backend (or local disk for development). See
+Requirements: Python 3.10+, a POSIX host (the durable monitoring spool uses
+`fcntl` locking), a reachable Redis 6+, and a configured storage backend (or
+local disk for development). See
 [docs/02_configuration.md](docs/02_configuration.md) for environment
 variables.
 
@@ -140,7 +150,7 @@ df, status, message = dr.execute(role_name="superadmin", engine=engine.AUTO)
 
 ## Demos
 
-The package ships two runnable demos under `supertable.demo`:
+The package ships three runnable demos under `supertable.demo`:
 
 ```bash
 # Numbered tutorial — runs the full lifecycle end-to-end.
@@ -152,9 +162,13 @@ python -m supertable.demo.quickstart
 supertable-demo-webshop-generate    # build ~1.2M rows on disk
 supertable-demo-webshop-load        # load them into SuperTable
 supertable-demo-webshop-topup       # continuous incremental refresh
+
+# Medical-center finance demo (raw → staging → mart, fully synthetic).
+supertable-demo-medcenter-run       # generate → load → transform → tests →
+                                    # accounting export → idempotency proof
 ```
 
-Both demos are also runnable as module steps. Examples:
+All demos are also runnable as module steps. Examples:
 
 ```bash
 python -m supertable.demo.quickstart.s01_01_01_create_super_table
@@ -171,7 +185,7 @@ script index.
 
 - **Versioned tables** with snapshot isolation, upsert (`overwrite_columns`),
   soft deletes (`delete_only=True`), schema evolution, and staleness filtering
-- **DuckDB query engine** — embedded, zero-copy reads from object storage
+- **DuckDB and IslandDB query engines** — embedded reads with adaptive selection
 - **Spark SQL via Thrift** — for queries exceeding DuckDB memory limits
 - **RBAC** — role types (superadmin, admin, writer, reader, meta) with
   row-level and column-level security enforced through view chains
@@ -202,13 +216,15 @@ See [docs/00_index.md](docs/00_index.md) for the full table of contents.
 | 06 | [Data Writer](docs/06_data_writer.md) | Write pipeline, locking, dedup, tombstones |
 | 07 | [Ingestion & Pipes](docs/07_ingestion.md) | Staging areas, automated ingestion pipes |
 | 08 | [Distributed Locking](docs/08_locking.md) | Redis locks, file locks, deadlock prevention |
-| 09 | [Query Engine](docs/09_query_engine.md) | DuckDB Lite/Pro, Spark SQL, auto selection |
+| 09 | [Query Engine](docs/09_query_engine.md) | DuckDB, IslandDB, Spark SQL, auto selection |
 | 10 | [Data Reader](docs/10_data_reader.md) | Read facade, snapshot history, view chain |
 | 11 | [RBAC & Access Control](docs/11_rbac.md) | Roles, users, row/column security |
 | 12 | [Audit Logging](docs/12_audit.md) | SHA-256 hash chain, DORA/SOC 2, SIEM |
 | 13 | [Table Mirroring](docs/13_mirroring.md) | Delta Lake, Iceberg, Parquet export |
 | 14 | [Monitoring](docs/14_monitoring.md) | Metrics writer, structured logging |
 | 15 | [Python SDK](docs/15_python_sdk.md) | Core classes, demos, example index |
+| 16 | [Redis Key Layout](docs/16_redis_layout.md) | Canonical key namespaces and validation rules |
+| 17 | [Privileged Audit Worker](docs/17_privileged_audit_worker.md) | Supervised deployment, durability preflight, safe trimming, probes, failure policy |
 
 ---
 

{supertable-2.4.0 → supertable-2.5.3}/README.md

@@ -2,7 +2,7 @@
 
 ![Python](https://img.shields.io/badge/python-3.10%2B-blue)
 ![License](https://img.shields.io/badge/license-FSL--1.1--ALv2-orange)
-![Version](https://img.shields.io/badge/version-2.0.0-brightgreen)
+![Version](https://img.shields.io/badge/version-2.5.1-brightgreen)
 
 **SuperTable — versioned data lake library for SQL analytics.**
 
@@ -23,8 +23,9 @@ pip install "supertable[gcp]"         # Google Cloud Storage
 pip install "supertable[all]"         # everything
 ```
 
-Requirements: Python 3.10+, a reachable Redis 6+, and a configured storage
-backend (or local disk for development). See
+Requirements: Python 3.10+, a POSIX host (the durable monitoring spool uses
+`fcntl` locking), a reachable Redis 6+, and a configured storage backend (or
+local disk for development). See
 [docs/02_configuration.md](docs/02_configuration.md) for environment
 variables.
 
@@ -97,7 +98,7 @@ df, status, message = dr.execute(role_name="superadmin", engine=engine.AUTO)
 
 ## Demos
 
-The package ships two runnable demos under `supertable.demo`:
+The package ships three runnable demos under `supertable.demo`:
 
 ```bash
 # Numbered tutorial — runs the full lifecycle end-to-end.
@@ -109,9 +110,13 @@ python -m supertable.demo.quickstart
 supertable-demo-webshop-generate    # build ~1.2M rows on disk
 supertable-demo-webshop-load        # load them into SuperTable
 supertable-demo-webshop-topup       # continuous incremental refresh
+
+# Medical-center finance demo (raw → staging → mart, fully synthetic).
+supertable-demo-medcenter-run       # generate → load → transform → tests →
+                                    # accounting export → idempotency proof
 ```
 
-Both demos are also runnable as module steps. Examples:
+All demos are also runnable as module steps. Examples:
 
 ```bash
 python -m supertable.demo.quickstart.s01_01_01_create_super_table
@@ -128,7 +133,7 @@ script index.
 
 - **Versioned tables** with snapshot isolation, upsert (`overwrite_columns`),
   soft deletes (`delete_only=True`), schema evolution, and staleness filtering
-- **DuckDB query engine** — embedded, zero-copy reads from object storage
+- **DuckDB and IslandDB query engines** — embedded reads with adaptive selection
 - **Spark SQL via Thrift** — for queries exceeding DuckDB memory limits
 - **RBAC** — role types (superadmin, admin, writer, reader, meta) with
   row-level and column-level security enforced through view chains
@@ -159,13 +164,15 @@ See [docs/00_index.md](docs/00_index.md) for the full table of contents.
 | 06 | [Data Writer](docs/06_data_writer.md) | Write pipeline, locking, dedup, tombstones |
 | 07 | [Ingestion & Pipes](docs/07_ingestion.md) | Staging areas, automated ingestion pipes |
 | 08 | [Distributed Locking](docs/08_locking.md) | Redis locks, file locks, deadlock prevention |
-| 09 | [Query Engine](docs/09_query_engine.md) | DuckDB Lite/Pro, Spark SQL, auto selection |
+| 09 | [Query Engine](docs/09_query_engine.md) | DuckDB, IslandDB, Spark SQL, auto selection |
 | 10 | [Data Reader](docs/10_data_reader.md) | Read facade, snapshot history, view chain |
 | 11 | [RBAC & Access Control](docs/11_rbac.md) | Roles, users, row/column security |
 | 12 | [Audit Logging](docs/12_audit.md) | SHA-256 hash chain, DORA/SOC 2, SIEM |
 | 13 | [Table Mirroring](docs/13_mirroring.md) | Delta Lake, Iceberg, Parquet export |
 | 14 | [Monitoring](docs/14_monitoring.md) | Metrics writer, structured logging |
 | 15 | [Python SDK](docs/15_python_sdk.md) | Core classes, demos, example index |
+| 16 | [Redis Key Layout](docs/16_redis_layout.md) | Canonical key namespaces and validation rules |
+| 17 | [Privileged Audit Worker](docs/17_privileged_audit_worker.md) | Supervised deployment, durability preflight, safe trimming, probes, failure policy |
 
 ---
 

supertable-2.5.3/pyproject.toml

@@ -0,0 +1,116 @@
+[build-system]
+requires = ["setuptools>=83.0.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "supertable"
+version = "2.5.3"
+description = "SuperTable — versioned data lake library for SQL analytics on Parquet + Redis."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "LicenseRef-FSL-1.1-ALv2"
+license-files = ["LICENSE"]
+authors = [{ name = "Levente Kupas", email = "lkupas@kladnasoft.com" }]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3 :: Only",
+  "Operating System :: POSIX"
+]
+
+# Read runtime deps from requirements.txt to keep a single source of truth
+dynamic = ["dependencies"]
+
+[tool.setuptools.dynamic]
+dependencies = { file = ["requirements.txt"] }
+
+[project.optional-dependencies]
+# Storage backends (pick what you deploy on)
+# botocore 1.35.2 is the first release whose PutObject model exposes
+# IfNoneMatch, required by immutable conditional publication.
+s3    = ["boto3>=1.35.2,<2.0"]
+minio = ["minio>=7.2,<8.0"]
+azure = [
+  "azure-storage-blob>=12.26.0,<13.0",
+  "azure-identity>=1.17,<2.0",
+]
+gcp   = ["google-cloud-storage>=3.1.0,<4.0"]
+all-cloud = [
+  "boto3>=1.35.2,<2.0",
+  "minio>=7.2,<8.0",
+  "azure-storage-blob>=12.26.0,<13.0",
+  "azure-identity>=1.17,<2.0",
+  "google-cloud-storage>=3.1.0,<4.0",
+]
+
+# Everything
+all = [
+  "boto3>=1.35.2,<2.0",
+  "minio>=7.2,<8.0",
+  "azure-storage-blob>=12.26.0,<13.0",
+  "azure-identity>=1.17,<2.0",
+  "google-cloud-storage>=3.1.0,<4.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/kladnasoft/supertable"
+
+[project.scripts]
+supertable-privileged-audit-worker = "supertable.audit.privileged_worker:main"
+supertable-redis-recovery = "supertable.recovery.redis_rebuild:main"
+supertable-demo-quickstart       = "supertable.demo.quickstart.controller:main"
+supertable-demo-webshop-generate = "supertable.demo.webshop.generate:main"
+supertable-demo-webshop-load     = "supertable.demo.webshop.load:main"
+supertable-demo-webshop-topup    = "supertable.demo.webshop.topup:main"
+supertable-demo-medcenter-generate  = "supertable.demo.medcenter.generate:main"
+supertable-demo-medcenter-load      = "supertable.demo.medcenter.load:main"
+supertable-demo-medcenter-transform = "supertable.demo.medcenter.transform:main"
+supertable-demo-medcenter-quality   = "supertable.demo.medcenter.quality:main"
+supertable-demo-medcenter-export    = "supertable.demo.medcenter.export_accounting:main"
+supertable-demo-medcenter-run       = "supertable.demo.medcenter.run:main"
+
+[tool.setuptools]
+include-package-data = true
+
+[tool.setuptools.package-data]
+"supertable.demo" = ["README.md"]
+"supertable.demo.medcenter" = ["annotations.json", "demo_queries.md"]
+
+[tool.setuptools.packages.find]
+include = ["supertable", "supertable.*"]
+exclude = [
+  "supertable.tests", "supertable.tests.*",
+  "supertable.*.tests", "supertable.*.tests.*",
+  "supertable.benchmarks", "supertable.benchmarks.*",
+  "supertable.*.benchmarks", "supertable.*.benchmarks.*",
+]
+
+[tool.pytest.ini_options]
+asyncio_default_fixture_loop_scope = "function"
+markers = [
+  "integration: requires an explicitly configured disposable external service",
+]
+
+[tool.ruff.lint]
+# Start with correctness rules that can never be waived by formatting taste:
+# syntax errors, invalid control flow, and undefined names. Broader rule groups
+# can be enabled incrementally without weakening this release-blocking baseline.
+select = ["E9", "F63", "F7", "F82"]
+
+[tool.mypy]
+python_version = "3.10"
+files = ["supertable"]
+# Tests and benchmarks have their own runtime gates.  Every shipped production
+# module (including demos/console entry points) remains inside the mypy target.
+exclude = [
+  "^supertable/(?:[^/]+/)*(?:tests|benchmarks)/",
+]
+follow_imports = "skip"
+ignore_missing_imports = true
+# Keep the debt baseline reproducible across the minimum/latest dependency
+# jobs; runtime compatibility is exercised by pytest, while this gate checks
+# SuperTable's own modules against one pinned mypy/typeshed version.
+no_site_packages = true
+check_untyped_defs = true
+incremental = false
+show_error_codes = true
+warn_unused_ignores = true

{supertable-2.4.0 → supertable-2.5.3}/requirements.txt

@@ -1,17 +1,21 @@
 # -------- Core data stack --------
 numpy>=1.26,<3.0
 pandas>=2.1,<2.3
-pyarrow>=16,<19
+pyarrow>=23.0.1,<24
 pytz>=2023.3
 tzdata>=2023.3
+apscheduler>=3.11.3,<4.0
 
 # -------- Engine / query / utils --------
-duckdb>=1.1,<2.0
-polars>=1.6,<2.0
-sqlglot>=26,<27
+duckdb>=1.5.1,<2.0
+polars>=1.36.1,<2.0
+sqlglot>=26.33,<27
 colorlog>=6.9,<7.0
-python-dotenv>=1.0,<2.0
-setuptools>=68
+tqdm>=4.66.3,<5.0
+python-dotenv>=1.2.2,<2.0
+starlette>=1.3.1,<2.0
+httpx>=0.27.1,<1.0
+cryptography>=50.0.0,<51
 
 # -------- Catalog / locking --------
 redis>=5.0,<6.0

supertable-2.5.3/setup.py

@@ -0,0 +1,6 @@
+"""Compatibility shim for legacy tooling; metadata lives in pyproject.toml."""
+
+from setuptools import setup
+
+
+setup()

supertable-2.5.3/supertable/__init__.py

@@ -0,0 +1,127 @@
+"""SuperTable — versioned data lake library for SQL analytics.
+
+SuperTable stores structured data as versioned Parquet snapshots on object
+storage (S3, MinIO, Azure Blob, GCP Cloud Storage, or local disk), keeps
+metadata and locks in Redis, and queries everything through DuckDB or
+IslandDB, with Spark SQL for fleet-scale workloads.
+
+Quick reference
+---------------
+
+>>> from supertable import SuperTable, DataWriter, DataReader, engine
+>>> st = SuperTable(super_name="example", organization="my-org")
+>>> dw = DataWriter(super_name="example", organization="my-org")
+>>> _, _, inserted, deleted = dw.write(
+...     role_name="superadmin",
+...     simple_name="facts",
+...     data=arrow_table,
+...     overwrite_columns=["day"],
+... )
+>>> dr = DataReader(super_name="example", organization="my-org",
+...                 query="SELECT count(*) FROM facts")
+>>> df, status, message = dr.execute(role_name="superadmin", engine=engine.AUTO)
+
+See the ``supertable.demo`` package for runnable end-to-end demos and the
+project documentation for the full API surface.
+"""
+
+__version__ = "2.5.3"
+
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+# This public alias predates the ``supertable.engine`` implementation package.
+# Importing the tiny enum after the now-lazy package is initialized ensures
+# later imports of engine submodules cannot replace ``supertable.engine`` with
+# the package object and break ``engine.AUTO``.
+from supertable.engine.engine_enum import Engine as engine
+
+
+# Importing the package is intentionally cheap and side-effect-free.  The
+# public convenience exports are loaded on first use (PEP 562) so importing
+# ``supertable`` does not import PyArrow/Polars, connect to Redis, probe the
+# filesystem, or initialise an application home.
+_LAZY_EXPORTS = {
+    "SuperTable": ("supertable.super_table", "SuperTable"),
+    "SimpleTable": ("supertable.simple_table", "SimpleTable"),
+    "DataWriter": ("supertable.data_writer", "DataWriter"),
+    "DataReader": ("supertable.data_reader", "DataReader"),
+    "query_sql_stream": ("supertable.data_reader", "query_sql_stream"),
+    "query_odata_sql_stream": (
+        "supertable.data_reader", "query_odata_sql_stream",
+    ),
+    "query_sql_policy_fingerprint": (
+        "supertable.data_reader", "query_sql_policy_fingerprint",
+    ),
+    "MetaReader": ("supertable.meta_reader", "MetaReader"),
+    "list_supers": ("supertable.meta_reader", "list_supers"),
+    "list_tables": ("supertable.meta_reader", "list_tables"),
+    "Staging": ("supertable.staging_area", "Staging"),
+    "SuperPipe": ("supertable.super_pipe", "SuperPipe"),
+    "RedisCatalog": ("supertable.redis_catalog", "RedisCatalog"),
+    "RoleManager": ("supertable.rbac.role_manager", "RoleManager"),
+    "UserManager": ("supertable.rbac.user_manager", "UserManager"),
+    "SupertableLookupError": ("supertable.errors", "SupertableLookupError"),
+    "SuperTableNotFoundError": ("supertable.errors", "SuperTableNotFoundError"),
+    "TableNotFoundError": ("supertable.errors", "TableNotFoundError"),
+}
+
+
+def __getattr__(name: str) -> Any:
+    target = _LAZY_EXPORTS.get(name)
+    if target is None:
+        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+    module_name, attribute = target
+    value = getattr(import_module(module_name), attribute)
+    globals()[name] = value
+    return value
+
+
+def __dir__() -> list[str]:
+    return sorted(set(globals()) | set(_LAZY_EXPORTS))
+
+
+if TYPE_CHECKING:
+    from supertable.data_reader import (
+        DataReader,
+        query_odata_sql_stream,
+        query_sql_policy_fingerprint,
+        query_sql_stream,
+    )
+    from supertable.data_writer import DataWriter
+    from supertable.errors import (
+        SuperTableNotFoundError,
+        SupertableLookupError,
+        TableNotFoundError,
+    )
+    from supertable.meta_reader import MetaReader, list_supers, list_tables
+    from supertable.rbac.role_manager import RoleManager
+    from supertable.rbac.user_manager import UserManager
+    from supertable.redis_catalog import RedisCatalog
+    from supertable.simple_table import SimpleTable
+    from supertable.staging_area import Staging
+    from supertable.super_pipe import SuperPipe
+    from supertable.super_table import SuperTable
+
+__all__ = [
+    "__version__",
+    "SuperTable",
+    "SimpleTable",
+    "DataWriter",
+    "DataReader",
+    "query_sql_stream",
+    "query_odata_sql_stream",
+    "query_sql_policy_fingerprint",
+    "engine",
+    "MetaReader",
+    "list_supers",
+    "list_tables",
+    "Staging",
+    "SuperPipe",
+    "RedisCatalog",
+    "RoleManager",
+    "UserManager",
+    "SupertableLookupError",
+    "SuperTableNotFoundError",
+    "TableNotFoundError",
+]